Reranking Models: The Cheapest RAG Quality Upgrade

Last year, the author revamped a support chatbot handling 40,000 help center and internal wiki chunks in a vector database. Retrieval used pure dense vector search, but answers were correct only about one-third of the time. Adding a reranker between retrieval and generation increased accuracy from 61% to 84%, with costs under $40 that month.

Many teams assume poor RAG answers indicate a need for better embedding models, larger LLMs, or new vector databases. However, often a cheap second pass over retrieved chunks is sufficient. This article explains reranking, its cost, how to measure the improvement, and when to skip it. Most Retrieval Augmented Generation (RAG) systems use bi-encoders, compressing queries and document chunks into single vectors and returning the closest candidates.

While fast, bi-encoders are better at recall, not precision. Negation, qualifiers, part numbers, and differences in phrasing like "refund policy for annual plans" vs. "refund policy for monthly plans" produce nearly identical vectors, leading to wrong chunks being ranked first. To address this, reranking splits retrieval into two stages: recall and precision.

Stage one uses fast bi-encoder search to pull top 50 to 100 candidates, while stage two scores each candidate with a reranker, a cross-encoder that reads the query and candidate together to produce a relevance score. Self-hosted rerankers cost only fractions of a GPU hour and add minimal latency. The author recommends hybrid retrieval, combining BM25 keyword search with dense vector search to better handle exact string matches. Implementing reranking adds latency but improves answer accuracy at a relatively low cost.