Prompt injection is a trust-boundary bug in the retrieval pipeline

Prompt injection represents a vulnerability in the retrieval pipeline of retrieval-augmented applications. These applications combine retrieved text with instructions and then send both to a language model. If the retrieved text was crafted by an attacker, the model receives instructions alongside its operator-provided prompts. This blurs the boundary between trusted and untrusted data, as the attacker's text arrives with the same authority as the context it was retrieved alongside.

In traditional applications, untrusted data undergoes parsing, validation, and quotation. The data does not become executable. SQL injection is mitigated by separating the query from the data, enforced by the database rather than relying on developer policies. Similarly, language models lack this separation by default, as their input is a token stream where instructions and content are indistinguishable.

Any text within the context window can influence behavior, and a document containing a directive is indistinguishable from one that merely mentions it.

The risk associated with this vulnerability is proportional to the application's capabilities after the model produces output. Retrieval systems answering questions over a corpus can leak sensitive content or return incorrect information. However, systems with tool access pose a more severe threat. If the model can execute actions such as calling APIs, sending messages, writing to databases, or executing code, an injected instruction followed by the model with operator privileges represents a significant violation of the user's session.

To mitigate this exposure, retrieved content should be treated as data, wrapped in a structure the model is trained to treat as untrusted, while instructions are kept separate. This approach reduces exposure but does not eliminate the risk entirely, as behavior under adversarial content remains probabilistic. Tool permissions should be constrained to the narrowest scope necessary for the use case, scoped to the current user rather than the application's service identity.

Actions that change state, send data, or incur costs should require confirmation from a human or an independent check, preventing the model from acting on injected instructions without human intervention.

Validating outputs against a schema before acting can catch garbled output and some attempts to redirect tool calls. Restricting the runtime's network access to only necessary addresses limits exfiltration risks even if the model is manipulated. However, detecting injected instructions is challenging due to the shared channel for both the carrying surface and the attack.

As a result, design should focus on containing the damage rather than detecting the injection itself. Logging the model's actions with tools, including arguments, user sessions, and retrieved documents, provides valuable evidence in the event of an incident. When designing features with tool access, consider the worst action an attacker could take if the retrieved content were crafted by an adversary; this determines the risk and informs the tool permission model.