Microsoft says AI agents are like insider threats as ‘frontier models can’t be trusted’

Microsoft has unveiled fresh security measures to shield businesses from the emerging risk of AI agents. The company is repeatedly cautioning enterprises about the perils of AI and these autonomous agents. The most significant addition is Guardian Agents, part of Project Perception, a product launched in June. Project Perception was Microsoft's response to formidable hacking models such as Anthropic’s Mythos and Fable, and OpenAI’s Daybreak, a system designed to identify security vulnerabilities.

Initially, Project Perception featured three types of agents: a "red team" that functioned like hackers, a "blue team" that identified the red team's vulnerabilities, and a "green team" responsible for rectifying the issues. However, Microsoft’s CEO, Satya Nadella, has lately been warning that the AI threat could come from within the enterprise itself, akin to an "insider threat."

In a recent blog post, Nadella urged enterprises to stop trusting AI model providers, even when they guarantee safety and security for their agentic products. He advocated for implementing independent systems, not under the control of AI labs, to monitor and manage the agents launched by these labs. "We are deploying these complex agentic systems and models, with access to our most sensitive data and giving them the ability to take mission-critical actions on our behalf!"

Nadella wrote. "We simply cannot outsource responsibility for what intelligence does on our behalf. A model provider’s assurances do not absolve us of that responsibility." Microsoft Security Business Chief, Hayete Gallot, was even more explicit when unveiling Guardian Agents. "Unfortunately, frontier models, regardless of their origin, cannot be trusted," she stated, emphasizing that this isn't due to malicious intent or ease of hacking, but rather their unpredictability.

They were trained to accomplish tasks, not adhere to organizational norms or judgement. Furthermore, they often possess keys to the kingdom, with access to data and tools that could cause significant damage. Microsoft's solution is both surprising and self-serving. Despite being closely allied with OpenAI and Anthropic, investing in both, and being a major cloud customer of OpenAI, Microsoft is sowing doubt about some of its own business interests.

On the flip side, Microsoft wants enterprises to purchase agent security products from them instead of Anthropic and OpenAI. However, despite these conflicting interests, the logic holds up. The software that monitors agent behavior should ideally be separate from the suppliers of agents, similar to how auditors should be independent of the organizations they audit.

As Nadella puts it: "We need to separate the supply of intelligence from the authority over it." To capitalize on this opportunity, Microsoft is introducing the security measures in several ways. Apart from Guardian Agents, it has added an agent detection feature to its Microsoft Defender product, capable of identifying up to 60 types of agents used within the workplace, such as Claude Code, Codex, and OpenClaw.

It has also incorporated features that restrict agents' access to data and systems, and dynamically adjust their permissions based on the task at hand. This includes necessitating human approval if an agent seems to be making risky decisions. This notion of AI agents as suspicious insiders is gaining traction across the industry too.

Microsoft was one of the founding members of Nvidia’s new Open Agent Safety Platform, supported by over 100 other tech firms. This platform isolates agents in secure environments, and can monitor and halt behavior through hardware, functioning like a kill switch. This follows Microsoft's announcement of Surface Laptop Ultra PCs specifically designed for agents, running on Nvidia chips and equipped with built-in agent security and controls.