RAG Context Budget: Pack Evidence Without Burying the Answer

When a Retrieval Augmented Generation (RAG) answer is inaccurate, the common response is to increase the top_k value. This means retrieving more chunks, which can feel safer because more retrieved chunks seem safer. However, in production environments, this approach can have adverse effects. While a large context window provides capacity, it does not constitute an effective retrieval strategy.

The input should be viewed as a context budget, where reservations must be made for the answer and the retrieved evidence should only be admitted if it justifies its token cost. This guide outlines a vendor-neutral policy that can be implemented with any vector store and model API.

Retrieval and generation serve different purposes, so the confusion between retrieval recall and prompt admission is a mistake. A retriever should create a broad candidate set to prevent missing a good passage, whereas the prompt assembler should construct a small, diverse, and attributable evidence set that the model can utilize. These are not the same top_k values.

For instance, a support assistant may retrieve 30 candidates for recall safety, rerank the best 12, and then provide four concise, non-overlapping sections to the model. Conversely, a policy question may require the most recent authoritative policy alongside an exception. When a "list every affected region" question is asked, more coverage might be necessary, and a different answer format required.

Sending all candidates to the model generates four issues: attention dilution, unstable cost and latency, crowded-out answers, and weak debugging capabilities.

Recent retrieval systems focus on selecting useful highlights instead of entire pages, which is logical; however, this does not eliminate the need for an explicit policy about what enters the generation prompt. The budget should be defined before tuning chunks. Start from the smallest context you want to support reliably, not the model's maximum window.

Reserve fixed and variable space first: for safety rules, format, tool policy, instructions, conversation relevant turns, and user constraints. Then, allocate evidence retrieved passages and metadata to compete fairly with the answer. This creates an equation: evidence_budget = input_limit - instruction_tokens - conversation_tokens - answer_reserve - safety_margin. Use the provider's tokenizer for the final count.

A rough estimate is acceptable for initial candidate selection, but remember that words do not equal tokens, and chat-message formatting is not free. The budget is also a product decision. If the user asks for comparisons with sources, allocate enough output for distinctions and citations. If the task returns a short JSON classification, the answer reserve can be smaller, as each task has its own requirements.

An effective context packer has four gates: retrieving candidates broadly, applying filters for tenant, access-control, language, and freshness before similarity search, reranking against the actual question using cross-encoders, rules, or small models, removing redundancy by keeping the best representative evidence with its source identity, and packing the remaining token budget.

Prefer authority, answer coverage, recency, and source diversity over score alone. This means a lower-scoring chunk may be admitted if it fills a gap in a multi-part question, while high-scoring duplicates should be excluded once their facts are already represented.

Having a clear query class is crucial for determining the packing rule. A small classifier can help choose a policy, and logging the decision is essential. Begin with deterministic routing based on evident cues and test data. If necessary, a classifier can later be used, but its output must remain observable and easily overridden.

A wrong class should return a narrower answer, ask a question, or disclose limitations rather than stuffing every result into the prompt. A simple packing policy in Python can be implemented, leaving a fixed answer reserve, rejecting near-duplicates, and providing a small diversity allowance for each source.