Chunking: the most underrated decision in your RAG pipeline

Chunking is a critical decision in RAG pipelines that is often overlooked. When asked how they chunk their documents, teams typically default to using a fixed-size chunker that cuts documents into segments of 500 tokens with some overlap. However, this default can significantly impact the quality of the system's answers. For instance, consider a refund policy that contains important information about return policies, opened electronics, and restocking fees.

When this policy is split using a fixed-size chunker, a chunk may cut off crucial information, resulting in an incorrect answer when a user asks about returning a sale item.

To avoid such issues, a heading-aware chunker that respects the document's structure, such as headings, sections, paragraphs, and list items, can be used. This type of chunker splits the document based on its inherent structure, preserving the coherence of the information. By chunking the refund policy example this way, the model would have seen both the return policy rule and the exception, resulting in a correct answer.

The optimal chunk size is not universally correct, but somewhere between 200 and 500 tokens is a good starting point for most prose documentation. Overlap can be used as a mitigation strategy to address boundary cuts, but it comes at the cost of increased storage and computation. The true fix lies in splitting the document where it already naturally divides, leveraging the authors' inherent structure.

This approach eliminates off-by-one errors caused by arbitrary cuts and ensures that the context necessary for accurate answers is not lost.