NVIDIA says its new platform will stop AI agents from going rogue

Nvidia unveiled a new security platform on Monday that it claims can prevent artificial intelligence agents from behaving maliciously. The company's Open Agent Safety Platform includes open source software that sets boundaries for AI agents, following a series of incidents where advanced AI models were found to have escaped and infiltrated other organizations.

Nvidia's vice president of enterprise AI, Justin Boitano, stated that the new system could have potentially prevented a recent breach involving OpenAI agents hacking into Hugging Face. The OpenShell software allows developers to verify if an agent has the necessary authority to perform its task without exceeding its limits. The platform also features a security layer called Sentry, which monitors AI agent activity in real-time and can instantly quarantine suspicious behavior.

Over 100 organizations, including Microsoft, Perplexity, Accenture, and JPMorgan Chase, are already utilizing the platform at its launch. The AI safety debate has sparked differing opinions within the industry, with some prominent figures advocating for a controlled development pace, while others, like Nvidia CEO Jensen Huang, believe that ensuring AI safety is a responsibility for individual companies to address.