Long context vs RAG: why search when a model reads it all?
For long context vs RAG, start with full-document input when one document fits and you need only a few answers; test retrieval for repeated questions over a large collection. RAG reduces prompt size, but its usefulness depends on retrieving the evidence each answer needs. A model that can take 700,000 words can read the whole document, yet repeated input can still incur charges, with discounts…
When deciding between long context and Retrieval-Augmented Generation (RAG) for processing large documents, the choice depends on several factors, such as the size of the document, the number of questions being asked, and the importance of fact accuracy.
If the entire document can fit within the model's context window, which currently ranges from 128,000 to 1 million tokens, using long context can be advantageous. In this case, sending the whole document with each question allows the model to directly access all relevant information, potentially leading to higher accuracy, especially when the answer is located in the middle of the text.
However, this approach incurs costs based on the number of input tokens, and repeated questions can lead to additional charges. On the positive side, if a caching mechanism is enabled, repeated questions can reduce costs as the system can reuse previously processed tokens.
On the other hand, RAG offers a more efficient solution when dealing with large collections of documents or when answering repeated questions. In RAG, the system first performs a search step to identify a few relevant passages from the entire collection. These passages are then included in the prompt along with the question, effectively narrowing down the amount of text the model needs to process.
This approach reduces the prompt size, leading to lower input token costs, especially when caching is enabled. The main risk with RAG lies in the retrieval process itself; if the search algorithm fails to find the correct chunk containing the answer, the model will not have access to that information, potentially leading to inaccurate or incomplete responses.
In summary, long context is suitable when the whole document can be processed at once, and the answer is likely to be located within the context window. RAG, however, provides a more scalable solution for dealing with large collections of documents, as it minimizes the amount of text sent to the model by focusing on the most relevant passages retrieved through a search process.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.