The state axis: why agent benchmarks keep measuring amnesiac models

The claim that coding-agent scores only represent the model, not the model plus harness, is a recurring point. The same context-carryover rules, note convention, tool loop, and judge must be consistent for a valid comparison. Engrim (github.com/timgordontg/engrim) underscores that a third axis has been underrepresented: state. This local-first SQLite memory engine is utilized by Claude Code, Cursor, Windsurf, and Codex.

It's project-scoped, stores records and SQLite FTS, and persists decisions and rationale across sessions. Cross-session recall is its key promise.

However, the issue at hand is that current benchmarks rarely measure this state-dependent memory capacity. The standard approach involves loading a repository, placing you at an issue, and scoring the patch. No cursor rules, no memory file, the model starts fresh each time. This clean reproducibility is indeed valuable, but it creates a distorted measurement. The agent being evaluated in a cold-start scenario is significantly different from the one your team would use after a month of accumulated project memory.

The gap widens as an agent retains more memory across sessions. A 30-minute cold-start evaluation offers little insight into how the agent with six months of project decisions would perform in a real-world migration. This is akin to correlated judges scoring a single session: you've measured one blind spot and hailed it as the committee. You've measured one memory state (empty) and called it the model. The variable that truly drives production output - prior state - is the very one the evaluation artificially sets to zero.

The solution doesn't have to compromise reproducibility. Instead, we can parameterize it. Run the task three ways: empty context, a small hand-curated project memory, and a large one. Observe how the score varies. The tool with the most significant score improvement should be the one you run in a long-lived repository. This result is far more valuable than a hundred clean cold-start tables.

The real takeaway is that if your evaluation measures amnesiac models, its scores are a floor, and the gap to production output represents the true signal.