Designing a Production RAG Retrieval System for 500 Million Vectors

Atlas, a RAG product, started simple but has grown to handle 10 million vectors. As customer data and search traffic increased, Atlas needed a more robust retrieval system. The goal was to design a system that could handle 500 million vectors without compromising performance or reliability.

The current architecture involves chunking documents, generating embeddings, storing vectors with metadata, and using these vectors for retrieval. This setup works for a smaller corpus but faces challenges when scaling up.

One issue arises when trying to maintain a single search unit for the entire dataset. A 768-dimensional float32 vector takes about 3 KB, making the raw vector data for 500 million vectors roughly 1.5 TB. This would exceed the capacity of a single machine, making it difficult to justify keeping everything in one place.

To address this, Atlas implemented a shard system, where the corpus is split into smaller parts, each with its own local search index. This allows for more efficient use of resources and easier scaling as the dataset grows. Additionally, the hottest search state can remain in memory while larger vector data and index structures can be moved to cheaper storage like SSDs.

However, this shard-based approach also creates a new challenge. As more documents are uploaded throughout the day, the write path needs to modify the optimized index in real-time. This can interfere with the read path, which involves finding relevant vectors, fetching corresponding chunks, and providing them to the LLM.

In summary, as Atlas grows to handle 500 million vectors, it faces capacity limitations and the need to separate read and write paths. The solution involves splitting the corpus into shards, each with its own local search index, and optimizing storage for different types of data. This redesign allows Atlas to maintain performance and reliability at a much larger scale.