The Harness Is Not Intelligence: What Is Actually Improving in AI Agents?
A few months ago, I wrote about a feeling I still have today: AI models, and especially coding agents, no longer give me the same sense of huge leaps that they used to. I am not saying they are not improving. Newer models usually make fewer mistakes, follow instructions better, and sometimes solve problems that older versions could not. But it is becoming harder for me to feel those improvements…
The Harness Is Not Intelligence: What Is Actually Improving in AI Agents
AI models, especially coding agents, may not provide the same feeling of huge leaps in capability as before. While newer models make fewer mistakes, follow instructions better, and sometimes solve problems that older versions could not, it is becoming harder to perceive those improvements as a real jump in capability. Recently, a post argued that models are simply different in behavior, and what is good or bad is the harness controlling them. While part of this idea is true, taking it too far is a mistake.
The harness, which includes tools, context management, system prompts, planning, retries, and more, is crucial. Two different products can use the same model but have vastly different experiences depending on the harness. A single model can produce different results when used in different agents due to variations in context management, tool usage, error recovery, and information relevance.
The result of an agent is a combination of the model, harness, context, tools, instructions, and task. Changing any of these components will affect the outcome. For example, using different models in similar environments can lead to varying performance, even if the harness is practically the same. This is because the model itself can make mistakes, even if it understands the instruction and implements a correct alternative.
A recent example involved using Claude Code on an existing project. The model correctly replaced alert() with a modal, but later, it left an alert() in another part of the system. Although the harness (Claude Code) understood the instruction and implemented the correct alternative, it failed to maintain that decision consistently.
This example raises questions about whose fault it was: mine, the harness, or the agent. Ultimately, if the model has the instruction and still ignores it, we must accept that the model made a mistake.
Good harnesses can help compensate for a model's limitations, but they do not necessarily make the model smarter. Instead, they detect, prevent, or correct its mistakes. This progress is significant and should not be overlooked. However, measuring the true improvement of models can be challenging, as advances may come from harness improvements rather than model advancements.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
