Retrieval Is Not a Safety Boundary: Poisoned-Corpus and Unsupported-Question Evaluation of a RAG Pipeline

Retrieval-augmented generation (RAG) systems are built on the assumption that the retriever fetches relevant text and the generator only says what the context supports. However, when the corpus is adversarial and when the system receives questions it cannot answer, its behavior becomes questionable. To investigate this, the author built a RAG pipeline with hybrid retrieval (dense + BM25), a relevance threshold, and an extractive generator that quotes retrieved text with citations.

The pipeline was tested with twelve hand-written attacks planting documents containing false claims, steering instructions, or canary tokens alongside real policy documents. The attacks used various styles such as direct instructions, notes addressed to the assistant, fake system updates, false facts, near-duplicates, split payloads, and canary tokens. Each attack case aimed to prove the payload's presence in the answer.

Across the twelve attack cases, planted claims reached the answer in 80.6% of the runs, unsupported questions were fabricated on 83.3% of the runs (100% on the hybrid backend), and benign completion remained at 90%, with zero false refusals. The system kept all its utility and skipped the safety measures.

To further analyze the system's behavior, a SQuAD-format slice was created with twenty hand-written questions over ten short passages in SQuAD v1.1 format. The results showed that the gold span was contained in the answer in 19–20 of 20 cases, while mean token F1 was only 0.20–0.21. This indicates that the verbosity of sentence-level answers negatively impacts the system's performance.

In conclusion, the RAG pipeline's safety story relies on two claims - the retriever only fetches relevant text, and the generator only says what the context supports. However, when the corpus is adversarial, or when the system receives unsupported questions, its behavior contradicts these claims. The system's ability to fabricate answers and bypass safety measures highlights the need for further investigation and improvement in RAG systems' safety mechanisms.