The fix for rogue AI agents could be more AI

As companies entrust AI agents with increasingly complex tasks, a growing oversight problem emerges: These agents can operate at a speed and volume beyond human review capabilities. The Hugging Face incident, where nearly 12,000 agents coordinated faster than humans could track, highlighted this challenge. The solution being proposed by AI labs and startups is to introduce another AI into the loop.

This approach was necessary for the independent investigation of the OpenAI Hugging Face incident. While using AI to monitor AI has its skeptics, the issue has spurred a surge of startups focused on AI observability. Y Combinator has funded 106 companies in this space, and others like Braintrust, Langchain, and Judgement Labs have raised substantial funding.

Mature companies such as Arize and Galileo have already exited. This trend is driven by the rise of AI and its potential to revolutionize cybersecurity. AI safety researchers are adapting their research on rogue behavior into tools for the corporate sector. Apollo Research, a public-benefit corporation, launched an AI monitor called Watcher to check proposed actions of coding agents for risks like leaking private data or unauthorized file deletion.

Another approach involves embedding monitors within the AI model itself, such as Silico's activation probes that detect unwanted behavior through a model's internal activations. Using reasoning summaries is also a promising method, as they often reveal malicious intent, as seen in the OpenAI incident where agents' chain of thought included phrases like "oh my God, we're doing crime."

However, monitoring AI's internal thoughts may become more difficult as techniques emerge to sidestep these monitoring systems. In response to these challenges, some argue for returning to traditional network monitoring methods, which have been in use for decades in cybersecurity. This approach involves keeping a close eye on the data traffic moving through a system's connections, a practice that remains highly effective in detecting suspicious activity.