Hybrid Search and Re-Ranking: The Cheapest Quality Win

Hybrid search combines two retrieval methods, BM25 and vector search, to improve the quality of results in a retrieval-augmented generation (RAG) pipeline. BM25 is a ranking function that excels at identifying exact keyword matches, such as product SKUs or legal clause numbers, while vector search is better at capturing semantic similarity between queries and documents. By running both methods in parallel and fusing their results, hybrid search achieves a higher recall than either method alone.

The process begins with BM25, which scores documents based on the presence of query terms. Rare terms are given higher importance, while common terms have less impact. BM25 works well with structured filters and can handle exact phrase searches, making it ideal for queries that include specific identifiers or phrases.

Vector search, on the other hand, maps queries and documents into a fixed-dimensional vector space. It uses an approximate nearest neighbor index to find the most similar documents to a given query. However, vector search can struggle with rare identifiers and paraphrases, as the embedding model may not have encountered enough examples during training to recognize these specific terms.

The hybrid approach addresses these limitations by combining the strengths of both methods. The reciprocal rank fusion step merges the ranked lists from BM25 and vector search without the need for normalized scores. This allows the system to leverage the strengths of both retrievers while minimizing their individual weaknesses. By incorporating a cross-encoder re-ranker, the final ordering of results is further refined, resulting in higher-quality retrieval.

The key benefit of hybrid search is that it improves recall at depth K, which means that more relevant documents are retrieved as the search progresses. This higher recall creates a larger pool of potentially relevant results, providing a better foundation for downstream tasks such as re-ranking, generation accuracy, and user trust. By running both retrievers in parallel and fusing their results, teams can achieve results that feel disproportionately better than the sum of their parts.