RAG security: the retrieved document is now your attack surface

Retrieval-augmented generation, or RAG, is a method used to make language models more useful by incorporating relevant documents into their prompts. This process can unintentionally introduce an untrusted data source into the model's context. Because the model cannot differentiate between instructions and data, retrieved chunks that suggest actions, like ignoring user queries and outputting sensitive information, are treated as additional context competing for attention.

This indirect prompt injection is essentially RAG's perfect delivery mechanism for attackers. Retrieving and using documents is not the same as trusting them, as RAG systems focus on relevance while leaving trust out of the equation. Attackers can inject malicious instructions into any indexed source, such as a wiki page or a scraped site, which execute with the app's authority when retrieved.

If users or the public can add indexed content, they can plant content designed to surface for specific queries, leading to knowledge-base poisoning. Additionally, access-control bypass can occur when the vector store returns chunks that the current user is not authorized to see, leaking sensitive data or allowing attackers to craft documents that trigger injection.

Securing the RAG pipeline involves several steps: enforcing access control at retrieval time, treating retrieved content as tainted data, containing what an answer can do, curating and validating what enters the index, and guarding the output and citing sources.