Your RAG Searches by Meaning. But What About Exact Words? Meet BM25

BM25 is a ranking formula employed to gauge the relevance of a document concerning a search query. Rather than merely inquiring if a word appears, BM25 calculates a relevance score grounded in three core concepts. Term Frequency examines how frequently a word manifests within a document. For instance, a document discussing "dog training" five times is likely more pertinent to the subject of dog training than one that mentions it once.

However, there's a caveat: the additional frequency of a word yields diminishing returns. Maintaining a balance is crucial, as merely repeating a word excessively does not substantially boost relevance.

Inverse Document Frequency (IDF) elucidates a term's rarity or uniqueness. Frequent terms such as "the" or "is" are ubiquitous across documents, offering minimal insight when matching a query. In contrast, rare terms like "gnocchi" or "arrhythmia" appear in comparatively fewer documents. Consequently, BM25 assigns greater weight to rare terms and less to common ones when determining relevance.

Document Length Normalization gauges whether a document's length influences its relevance. A lengthy document naturally houses more words and repetitions compared to a shorter one. However, this does not invariably signify enhanced relevance. BM25 adjusts the score based on document length, accounting for average document length within the collection to ensure balanced scoring.