Freeze a Holdout Before You Quote a Coding-Agent Score
You are in a Friday ranking review. Someone pastes a table: Agent B hit 71 percent. Last week it was 58. The room relaxes. Then you ask whether the system prompt changed after the last failure dump. It did. Three times. The 71 is a fitted number. It is not a measurement. That is how coding-agent scoreboards rot. You inspect failures, you edit the prompt, you rerun the same tasks, you publish the…
You are participating in a ranking review on a Friday. A table is pasted, showing that Agent B achieved a 71% score, up from 58% the previous week. The room relaxes, and you inquire whether the system prompt altered after the last failure dump. It did, and this change occurred three times. The 71% figure is a fitted number, not a measurement.
This is how coding-agent scoreboards deteriorate over time. You examine failures, edit the prompt, rerun the same tasks, and publish the new pass rate. The worksheet has become the exam. To obtain a defendable number, you must freeze a holdout first and never tune against it. This article serves as a protocol, not a leaderboard.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
