I finally cut AI context bloat by 80% and the fix was embarrassingly simple

A few months ago, a deep-seated issue emerged in AI agents. These agents, after fetching, parsing and storing data, began pasting the entire raw tool response into subsequent model calls. Full JSON blobs, logs, metadata and everything else were included, causing the next step to slow down and become more expensive. The model started answering the easy part of the task while ignoring the important part.

This bug highlighted that most "context window problems" are not model issues, but state management problems. Once this concept was understood, the issue became more apparent across various AI orchestration systems. The solution was simple: do not replay raw tool output into every prompt. Instead, store full results in structured state outside the model and send only the necessary fields for the next step or the exact evidence needed for verification.

This change significantly improved the quality of agents, often more than simply switching to a different model. The flawed approach was to append the entire tool response to the prompt, filling it with unnecessary information, which forced the model to sift through the noise and waste resources. The recommended approach was to store the full result in application state and pass only the relevant fields to the next step.

While a large prompt might seem to give the model more context, it often leads to a poorly managed state store, increased latency, higher costs and a less focused model. The best practice is to use structured state outside the prompt for multi-step agents, with short recurring prompts for cleaner reasoning and better orchestration.