Two independent audits of Terminal Bench 4: 41 of 65, or 30 of 66?
Epoch AI published a benchmark review of Terminal-Bench 4.0.0 and gave it the verdict Flawed: 30 of 66 tasks (45.5%) carry publicly documented scoring defects. Separately, Preference Model ran its own audit of the same benchmark and reported that in 41 of the 65 tasks they ported, a model could do honest, correct work and still fail. Two groups, two counts, one benchmark. Neither number is wrong.…
Two separate audits of Terminal-Bench 4.0.0 yielded divergent results: Epoch AI found 30 out of 66 tasks (45.5%) with scoring defects, while Preference Model reported 41 out of 65 tasks (63.1%) where the grader enforced unclarified rules or allowed workarounds. Neither figure is incorrect, as they assess different aspects of the benchmark.
Epoch's review relied on a published rubric, identifying four categories of defects: reward hacking, answer leakage, false negatives, and false positives. Preference Model, conversely, counted tasks with grader-imposed rules not clearly stated in the instructions. Both audits also differed in the benchmark datasets used and the settings in which they were conducted.
Regardless of the methodology, the audits collectively illustrate that roughly half of the tasks in the benchmark contain issues with grading, hinting at a significant gap between model performance and the reliability of the grading system.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
