{
  "id": 6354992,
  "title": "The state axis: why agent benchmarks keep measuring amnesiac models",
  "url": "https://urgent.news/2026/09/09/the-state-axis-why-agent-benchmarks-keep-measuring-amnesiac-models",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-09T00:15:03.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/cole_halton_42f71d71b809b/the-state-axis-why-agent-benchmarks-keep-measuring-amnesiac-models-39f4"
  },
  "original_language": "en",
  "account": "The claim that coding-agent scores only represent the model, not the model plus harness, is a recurring point. The same context-carryover rules, note convention, tool loop, and judge must be consistent for a valid comparison. Engrim (github.com/timgordontg/engrim) underscores that a third axis has been underrepresented: state. This local-first SQLite memory engine is utilized by Claude Code, Cursor, Windsurf, and Codex. It's project-scoped, stores records and SQLite FTS, and persists decisions and rationale across sessions. Cross-session recall is its key promise.\n\nHowever, the issue at hand is that current benchmarks rarely measure this state-dependent memory capacity. The standard approach involves loading a repository, placing you at an issue, and scoring the patch. No cursor rules, no memory file, the model starts fresh each time. This clean reproducibility is indeed valuable, but it creates a distorted measurement. The agent being evaluated in a cold-start scenario is significantly different from the one your team would use after a month of accumulated project memory.\n\nThe gap widens as an agent retains more memory across sessions. A 30-minute cold-start evaluation offers little insight into how the agent with six months of project decisions would perform in a real-world migration. This is akin to correlated judges scoring a single session: you've measured one blind spot and hailed it as the committee. You've measured one memory state (empty) and called it the model. The variable that truly drives production output - prior state - is the very one the evaluation artificially sets to zero.\n\nThe solution doesn't have to compromise reproducibility. Instead, we can parameterize it. Run the task three ways: empty context, a small hand-curated project memory, and a large one. Observe how the score varies. The tool with the most significant score improvement should be the one you run in a long-lived repository. This result is far more valuable than a hundred clean cold-start tables. The real takeaway is that if your evaluation measures amnesiac models, its scores are a floor, and the gap to production output represents the true signal.",
  "summary": "I keep hammering the point that any coding-agent score is model + harness, not model alone. Same context-carryover rules, same note convention, same tool loop, same judge, or the comparison is garbage. Engrim (github.com/timgordontg/engrim) is a useful reminder that there's a third axis I've been underselling: state. It's a local-first SQLite memory engine for Claude Code, Cursor, Windsurf,…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}