RAG vs. Direct Context: I Tested Both on Real Documents, Here's What Broke

I conducted a hands-on test comparing BGE-M3 + Qwen3 (RAG vs. direct-context answering) on real documents, including a research paper and a book. The goal was to determine if AI models are actually reading the document or merely pattern-matching to text that resembles the question. I built an open-source pipeline to generate separate answers for each approach.

For the research paper, both RAG and direct answers agreed on the topic of the paper, which presented a transformer-based bidirectional machine translation system for the legal domain. The RAG approach retrieved chunks from the document, while the direct answer read the raw text. The retrieval process was straightforward, using fixed-size chunking and plain cosine similarity without any fancy tricks.

However, when testing a full-length book, RAG failed to provide an accurate answer. It mistakenly identified a footnote in the dedication as the paper's main contribution, which was almost word-for-word a title of a blog post. This issue stemmed from the model not filtering out the front matter, such as dedications, acknowledgments, and footnotes, before processing the text. In contrast, the direct answer correctly identified the document as a book review and promotional content for a specific work.

On a more positive note, when asked a more specific question about the book, RAG performed exceptionally well. It accurately stated that the retrieved context did not include a relevant answer to the question about the main contribution of the paper. This demonstrated that RAG can work effectively when the retrieved context genuinely lacks the necessary information, as opposed to providing incorrect or misleading answers.