My RAG Pipeline Got Hijacked by Retrieved Text: An Accidental Prompt Injection

A report has emerged detailing accidental prompt injection of a retrieval-augmented generation (RAG) pipeline. The pipeline, which compares two methods of answering questions about a document, first searches for the most relevant snippets using BGE-M3, then feeds those snippets to an AI model for answer generation using Qwen3. The testing was conducted on a free Google Colab GPU.

The original pipeline encountered a retrieval bug in part 1, where a footnote buried in the book's dedication page was mistakenly used as the RAG answer instead of the actual content. To fix this, a general noise filter was implemented before text is turned into an embedding. This filter flags chunks that resemble tables of contents, indices, or footnotes based on specific signals like digit ratios, ellipsis patterns, and short lines. The general filter successfully prevented the noise from reaching the AI model.

However, an unexpected issue arose when the pipeline returned a single character '0' as the answer to the question "What is this document about?" This occurred after the noise filter passed the retrieved chunks. Investigation revealed that one of the top-ranked chunks contained a worked example from the book, which should have been filtered out by the noise filter. The pipeline's fixation on this single digit character suggests a potential bug in the reranking process.

The report highlights the importance of robust noise filtering and reranking mechanisms in RAG pipelines to prevent prompt injection attacks. The findings demonstrate the complexity of ensuring the accuracy and security of AI-driven question-answering systems, particularly when dealing with diverse document structures and content.