Context Engineering, Explained

What's emerging in the realm of artificial intelligence is a new approach to handling context, known as context engineering. As models' context windows expand to encompass hundreds of thousands of tokens, developers are shifting their focus from crafting a single, optimal prompt to treating every element the model encounters—system instructions, retrieved documents, tool results, conversation history, and memory—as part of a cohesive, deliberate budget.

The allure of a larger window can be tempting; one might be inclined to fill it with everything just in case. However, this approach can be counterproductive. Irrelevant context can dilute the model's attention, leading to a measurable decrease in accuracy, even if the correct answer is present somewhere within the window. More space doesn't automatically translate to better utilization of that space.

The actual contents of this context budget are determined by three key components: the system prompt, retrieved documents, tool results, conversation history, and memory. The system prompt, which outlines the role, constraints, and output format, is set once and then reused for every call. Retrieved documents, or RAG chunks, are the specific chunks pulled for a particular question, and relevance trumps volume here.

Tool results, whether from function calls or searches, often represent the most noisy part of the window. Conversation history, the prior turns of the model, should be pruned aggressively in long agent loops to avoid overcrowding the window with unnecessary information. Memory, the facts carried over from session to session, should be curated rather than a haphazard dump of every past interaction.

A common mistake in context engineering is pasting entire documents instead of just the relevant chunk. Another frequent error is letting tool output accumulate without pruning as a long agent loop progresses, potentially overwhelming the actual task. It's also a mistake to assume the model perceives everything with equal weight. Both position and relevance matter; the model doesn't just see what's present, but what's relevant.

Developers interested in building RAG (Retrieval-Augmented Generation) or agents beyond the demo stage should take note. If you find your system prompt growing longer while accuracy declines, it's often a sign that curation is needed, not a bigger model.