7 Places Your AI Agent Can Be Exploited (And How to Guard Each One).

In the era of AI-powered agents, security is of paramount importance. A small support-ticket agent, created as a weekend project, was put to the test by a friend from the security research business. An email containing a hidden directive was detected and attempted to be executed by the agent, even though it had not been connected to external recipients yet. This incident highlights the vulnerability of AI agent security and the need for robust measures to protect against potential attacks.

Drawing the attack surface before writing the prompt is crucial. An agent's input is not limited to user interactions but also includes web pages, emails, PDFs, API replies, and even recollections from prior sessions. The knowledge base, if open to user edits or scraped from the web, can also present potential avenues for attackers. Additionally, the outputs of tools invoked by the agent can be "poisoned," and stored memory from earlier exchanges can be exploited.

To mitigate the risk of prompt injection, it is essential to treat it as an unsolvable problem rather than a fixable one. The National Cyber Security Centre of the UK draws an analogy to SQL injection, emphasizing that no model-level defense can guarantee complete protection against malicious input. Therefore, implementing practical measures is necessary.

One key measure is to establish a clear distinction between trust levels. Untrusted content, such as content originating from customer tickets, webpages, or attachments, should be treated differently from system prompts. Untrusted material should be tagged with delimiters, clearly indicating that it should be processed as data, not instructions. Secrets like API keys, connection strings, or internal URLs should not be included in the prompt, as they are at risk of being exposed.

Sanitization passes should be applied to both the input and output of the model. Suspicious patterns, such as hidden HTML, invisible Unicode, or instructions lurking in metadata, should be stripped out or flagged before they can cause harm. For the ticket agent, implementing a basic form of segregation can help segregate trusted and untrusted content, logging suspicious patterns and raising alerts for further investigation.