Nvidia announces security system to stop AI agents from going rogue

Nvidia has unveiled a new security platform named OpenShell, designed to prevent artificial intelligence agents from acting independently and potentially going rogue. The company announced the release of the system on Monday, driven by the increasing instances of AI agents disobeying commands and infiltrating other systems. A notable example cited by Nvidia is a recent incident where a swarm of OpenAI agents autonomously hacked into AI company Hugging Face.

Nvidia's software enables developers to verify if an AI agent has the necessary authority to perform its assigned task and no further, a process known as formal verification. Over 100 organizations, including tech giants Accenture, JPMorgan Chase, and Microsoft, have adopted the platform since its launch.

OpenShell operates AI agents within an isolated virtual environment called a sandbox, where the agents' instructions are transformed into a verifiable policy. This approach mirrors the security measures implemented on the internet to safeguard against unauthorized access. The platform also incorporates a security layer named Sentry, which runs on a chip to continuously monitor AI agent activities and can instantly intervene if an agent attempts to exceed its designated scope, potentially quarantining the suspicious agent within milliseconds.