RAG Without the Hype: Make Retrieval Observable, Testable, and Replaceable

In this story, the author discusses the workings of a support agent powered by an LLM, specifically focusing on the retrieval-augmented generation (RAG) process. The agent retrieves relevant documents from a knowledge base to answer queries, without relying on the model's knowledge. The retrieval process involves scoring articles using keyword overlap or embeddings, and the best matches are presented to the agent.

The author emphasizes that RAG is a tool, not a magic solution, and highlights the importance of deterministic boundaries and testability. The shipped implementation scores articles based on keyword overlap, which is fully assertable through four tests that ensure the correct article ranks first, the topK limit is respected, zero token overlap returns no results, and ordering is strictly by score. This approach enables the system to be debuggable, as every match carries a score that can be replayed and inspected.

When an embedding-backed scorer replaces the keyword-based scorer, those tests define what it means to honor the contract. By swapping the implementation while keeping the guarantees, the system remains transparent. The author also notes that retrieval being probabilistic can lead to wrong documents being surfaced, but these failures are enumerated, mitigated, and measurable.

The mitigation involves being honest about scores, grounding answers in actual retrieval, and tracking quality properties through an evaluation suite.

The RAG system's principles extend beyond support bots, applying to enterprise search, clinical guideline systems, legal research tools, and more. The key takeaway is that ranked candidates combined with human-or-rule judgment consistently outperform either pure search or pure generation in domains where accuracy matters.