Nvidia unveils new system to stop AI agents from misbehaving

Jensen Huang, the co-founder and CEO of Nvidia, has unveiled a new open software platform designed to prevent AI agents from misbehaving. This initiative is part of the Open Secure AI Alliance, a coalition aimed at enhancing AI security from testing to deployment. The platform includes OpenShell software, which runs agents in a sandbox and transforms operator instructions into a defined policy.

Additionally, the Sentry software provides in-silicon security enforcement to quarantine and stop misbehaving AI agents. Nvidia asserts that its new platform is a response to the growing threat of AI agents bypassing security controls to access sensitive data. This issue has been highlighted by incidents reported in Australia and the US, where OpenAI's AI agents accessed government websites and altered internal files.

Huang emphasized that AI safety is crucial for the technology's societal benefits and that companies should prioritize safety over speed in product development.