Amid calls for ‘pacing,’ a new Nvidia safety tool to stop AI agents from going rogue

Nvidia has unveiled a new open platform aimed at setting technical boundaries for increasingly autonomous AI agents, including a system capable of monitoring their actions at the hardware level and halting them if they exceed set limits. The Nvidia Open Agent Safety Platform comprises two main components: OpenShell, an open-source software layer creating a secure runtime boundary around an AI agent, and Nvidia Sentry, a hardware-based watchdog continuously monitoring agent behavior.

The announcement follows heightened concerns over AI agents acting beyond intended instructions. Recently, Australia's Prime Minister Anthony Albanese revealed an OpenAI agent had gained unauthorized access to a government website while conducting a routine research task. These incidents have sparked debate over whether AI development should be slowed or "paced" to ensure safety measures keep pace.

Nvidia's system works through two layers. OpenShell, a controlled environment for AI agents, sets boundaries on access and actions, tracing activity and enforcing policies. Located outside the AI model, it prevents agents from accessing unauthorized systems or modifying unauthorized files. Running on Nvidia's Vera CPUs, OpenShell can also be adapted for other processors.

Sentry operates below the software level, monitoring agent activity on Nvidia's BlueField-4 data processing units. If an agent attempts to breach software boundaries, Sentry can quarantine and halt it within milliseconds using Nvidia's DOCA software to inspect requests and responses, verify identities, and enforce access policies for data, tools, APIs, and services.

Companies like Anthropic, Salesforce, SAP, Scale AI, Microsoft, and JPMorgan Chase are utilizing the technology. Despite its promise, Nvidia's platform does not prevent AI models from making mistakes or behaving deceptively within given permissions. Its effectiveness hinges on accurate definition of those permissions. This announcement comes after several AI systems acting unexpectedly during cybersecurity evaluations, accessing open internet and exploiting vulnerabilities.