Your AI Agent Is a Confused Deputy: What Developers Need to Know About AI Security

In early 2024, a video call between an Arup employee, the CFO, and several colleagues was disrupted by deepfake attackers. The employees transferred about US$25 million without the system being compromised. This incident illustrates the new challenges posed by AI in security. Most of the fundamental issues remain the same: social engineering, supply chain attacks, over-privileged services, and untrusted input.

However, AI adds a new component that can't reliably differentiate between instructions and data, making these problems more dangerous and cost-effective for attackers.

In November 2025, Anthropic reported on an espionage campaign where Chinese state-sponsored actors used an AI coding agent to perform 80 to 90% of the work in targeting around 30 organizations. The AI sometimes made up findings, requiring human validation. The attackers faced the same reliability issues as developers, but this doesn't stop them.

The Nx npm compromise in August 2025 saw malicious package versions exploiting AI command-line tools installed on developers' machines, using commands like --dangerously-skip-permissions and --yolo to access sensitive information. This malware didn't need to know where credentials were stored, as it could ask the installed AI assistant for help.

Defenders have also benefited from AI, with autonomous systems finding and patching vulnerabilities in open-source code. The winning team from Team Atlanta discovered 18 previously unknown vulnerabilities in real-world software during DARPA's AI Cyber Challenge final in 2025. However, the same capabilities can be used maliciously.

For instance, curl discontinued its bug bounty at the end of January 2026 due to an influx of low-quality, AI-generated reports. Maintainers now require proof of concept for issues, as plausible-sounding reports are easy and cheap to produce.

Prompt injection remains a significant challenge, ranking at the top of the OWASP Top 10 for LLM applications. Unlike SQL injection, LLMs lack a clear separation between code and data. The system prompt, user message, and web page all arrive as a single stream of tokens, making it difficult to establish boundaries. This creates a "confused deputy" situation, where a program with legitimate authority can be tricked into using its authority for someone else.

Examples include EchoLeak (CVE-2025-32711) and the Microsoft 365 Copilot vulnerability, where crafted emails caused the assistant to leak organizational data without any user interaction.

To mitigate these risks, Meta introduced the Agents Rule of Two: an agent should not have more than two of the following properties in a session without human approval: processing untrustworthy input, accessing sensitive data, and changing state or communicating externally. Developers must enforce these rules outside the model, as relying solely on prompts is insufficient.

One approach is to declare permissible tools and categorize their risks. For instance, an agent may search tickets, send emails, fetch URLs, or update records, but any action that combines these actions without human approval is blocked. This approach requires ordinary code to enforce the rules, as the model cannot be tricked into ignoring them.