Embeddings Turn Text Into Numbers. But There’s More Than One Way to Do It

Hello, I am Rijul, and I am creating LiveReview - an AI code review tool designed for business-critical systems. If you are familiar with Retrieval-Augmented Generation (RAG), you likely know about embeddings - when text is turned into numbers. However, there is more to it than that. There exist two main approaches for representing text for retrieval: sparse representations and dense embeddings.

Sparse representations work by assigning a value to each word in a dictionary based on its relevance to the sentence. Most sentences contain only a few words from this dictionary, resulting in most of the vector being zero. For example, using the words ['cat', 'dog', 'run', 'fast', 'food', 'eat', 'blue', 'sky', 'car', 'drive'], the sentence "The dog can run fast" can be represented as 0 1 1 1 0 0 0 0 0 0, showing which words are important.

However, this method struggles with synonyms, paraphrasing, and conversational queries, as it relies on exact word match. Dense embeddings, on the other hand, capture the semantic meaning of text. Instead of focusing on individual words, they represent a piece of text as a vector with many non-zero values. For instance, "The dog ran quickly" and "The puppy sprinted" can have embeddings that are close together because they convey a similar idea.

While dense embeddings are useful for semantic search, they may not be as effective when exact terms are crucial, such as product codes, exact names, error codes, or rare technical terms. Therefore, RAG systems often use both sparse representations and dense embeddings to leverage their respective strengths and overcome their limitations.