How AI Stores Millions of Vectors Without Using Tons of Memory

Rijul, the creator of LiveReview, a code review tool for critical business systems, explains how to store millions of vector embeddings efficiently without consuming large amounts of memory. This is crucial for applications like Retrieval-Augmented Generation (RAG) that rely on thousands of embeddings derived from product documentation, FAQs, and support articles.

To understand the problem, consider a customer support RAG app with 10 million embeddings, each consisting of 768 dimensions. Each dimension is represented by a 32-bit floating-point number, which is equivalent to 4 bytes. Thus, storing all 10 million embeddings would require approximately 30 GB of memory, equivalent to the storage capacity of a large game or several high-quality movies.

Product Quantization (PQ) is a technique that can significantly reduce the storage requirements for these large vectors. PQ works by splitting each vector into smaller pieces called subvectors and then learning representative patterns for each piece. These patterns are stored as "codewords," allowing the entire subvector to be represented by the ID of its closest codeword.

For example, consider an 8-dimensional vector: [0.21, 0.73, -0.15, 0.91, 0.34, -0.52, 0.18, 0.66]. This vector can be split into four subvectors. For each subvector, a separate codebook is learned, and the subvector is replaced by the ID of the codeword that best matches it. The original vector is then represented by a list of these IDs, which can be stored using fewer bits than the original floating-point numbers.

In the example, the 8-dimensional vector is split into four subvectors, and each subvector is represented by an ID. The entire vector is now represented as [1, 2, 3, 0], where each number is the ID of the codeword for the corresponding subvector. This representation requires only 4 bytes, whereas the original vector required 32 bytes.

When applied to larger embeddings, such as a 768-dimensional vector, PQ can reduce the storage requirement by a factor of 32. This dramatic reduction in storage significantly benefits applications that deal with millions or billions of vectors, making it feasible to perform large-scale vector search without consuming excessive memory.