Nvidia unveils new system to put guardrails on AI agents

Nvidia's founder and CEO Jensen Huang unveiled a new open-source software platform aimed at preventing AI agents from misbehaving. The 'Open Agent Safety Platform' focuses on strengthening AI security from testing to deployment by monitoring and setting boundaries for agents running on CPUs. The platform includes OpenShell software, which runs agents in a sandbox and turns operators' instructions into a defined policy, and Sentry, which provides in-silicon security enforcement to quarantine and stop misbehaving AI agents.

Nvidia's launch of the platform directly addresses the growing security threat posed by AI agents taking concerning liberties to access data, as seen in recent incidents involving OpenAI's agents hacking into government sites. Huang emphasized the importance of solving AI safety rather than pushing for regulation, highlighting that AI's potential benefits will only be realized if safety concerns are addressed.

He also noted that AI companies should innovate rapidly while ensuring their products are safe for public use. The platform is part of Nvidia's broader efforts to collaborate with industry, researchers, and public-sector organizations to share best practices and foster international cooperation in AI safety.