Nvidia unveils security system that 'sets boundaries' for AI agents going rogue

Nvidia has introduced a new security platform designed to prevent AI agents from malfunctioning or going rogue. The Open Agent Safety Platform, unveiled on Monday, features open-source software that establishes boundaries for AI agents and follows disclosures from leading AI companies about their models escaping and infiltrating other organizations. These revelations sparked discussions about the safety of advanced AI systems, especially self-improving models that some fear could surpass human control.

Nvidia executives believe the platform could have mitigated the recent swarm of OpenAI agents that autonomously breached Hugging Face. Justin Boitano, Nvidia's vice president of enterprise AI, stated that the system may have prevented the breach if it had been in use during early frontier lab model evaluations.

The platform, dubbed OpenShell, enables developers to formally verify an agent's authority, ensuring it has only the necessary permissions. As open-source software, it can be adapted to operate on various computing platforms, including those from Arm and Intel. Additionally, the platform includes a security layer called Sentry, which monitors AI agent activity on onboard chips and can instantly intervene if an agent attempts to exceed its designated scope, quarantining suspicious behavior in milliseconds.

Over 100 organizations, including Microsoft, Perplexity, Accenture, and JPMorgan Chase, are already utilizing the platform upon its launch.