RAG Evaluation: How to Know Your Retrieval Isn't Broken

Retrieval-Augmented Generation (RAG) has become a popular approach to combining language models with external knowledge sources. However, a recent incident highlighted how fluent output can hide broken retrieval. During a meetup demo, a presenter showcased a RAG system that generated fluent, well-cited answers. Yet, none of the attendees noticed that the retrieval layer was only returning the correct chunk about 30% of the time.

This example demonstrates the importance of evaluating both retrieval and generation separately, using metrics on controlled data. The metric taxonomy for RAG evaluation consists of retrieval metrics and generation metrics. Retrieval metrics assess whether the retriever is finding the right source, while generation metrics evaluate if the generated answer uses the retrieved context correctly.

The most crucial retrieval metric is hit rate, which measures the fraction of queries where the correct document is present in the top-k retrieved results. Other important retrieval metrics include Mean Reciprocal Rank (MRR), Precision@k, NDCG (Normalized Discounted Cumulative Gain), and Faithfulness. Generation metrics include Answer Relevancy, Context Precision, and Context Recall.

These metrics allow teams to identify which layer is losing quality and prioritize engineering efforts accordingly. Building a labelled evaluation set is essential for meaningful RAG evaluation. The recommended approach is to start with 100-200 real question-document pairs from logs, label the ground truth, and include hard cases such as multi-hop questions and exact codes.

Version control the evaluation set and iterate on the chunker, embedding model, or retriever to assess improvements. By following this metric taxonomy and building robust evaluation sets, teams can ensure their RAG systems are both fluent and accurate.