Testing Provenance-Based Controls Against Indirect Prompt Injection in AI Agents

Defenses against prompt injection are built on the assumption that malicious text can be identified through a classifier. However, a weekend project revealed this approach is flawed. The author developed a tool that never reads the attacker's words, catching attacks that more sophisticated detectors missed. Unfortunately, the tool also blocked legitimate user actions, highlighting a trade-off that the industry currently struggles to resolve.

The core issue lies in the fact that AI agents read and act upon untrusted content, such as documents, web pages, and tool outputs. Any malicious instructions can be hidden within this untrusted text, making it impossible to prevent attacks by simply detecting malicious wording. The author argues that the solution lies in tracking the provenance of data, or where the data originates, rather than attempting to decipher the content itself.

By distinguishing between user-provided and untrusted source text, the tool can enforce policies that prevent the execution of actions based on untrusted data. This approach requires no machine learning models or complex classifiers, relying instead on a simple algorithm to track the origin of data and enforce strict rules to prevent unauthorized actions.