Stop Benchmarking Coding Models by Vibes: A Repeatable 20-Task Harness You Can Run Tonight
Scroll DEV this week and you'll see the same pattern: agents everywhere, and everyone arguing about which model "codes better." One recent discussion made a sharp point — metrics from sub-agents aren't comparable to main-thread metrics — and the replies were full of people realizing their own comparisons were just vibes with extra steps. That stung, because it's true of most of us. We paste the…
This week, numerous articles in Scroll DEV have highlighted the trend of people comparing coding models based on their outputs, arguing about which model codes better. A recent discussion pointed out that metrics from sub-agents cannot be compared to main-thread metrics, and many replies revealed that people were indeed just using vibes to make their comparisons more complicated.
This realization is common among most of us. We simply paste the same prompt into two models, skim their outputs, and declare a winner. However, this is not an evaluation; it's a coin flip disguised as an assessment, which is not an accurate representation of the true value of the models.
In response to this issue, the author has created a small, repeatable harness consisting of 20 fixed coding tasks. The harness provides deterministic scoring where possible and structured human rubric scoring for tasks where determinism is not applicable. The entire setup requires just one Python file and one JSONL file, with no framework, database, or dashboard needed.
The purpose of this harness is to provide a repeatable evaluation of any OpenAI-compatible model without relying on public benchmarks that only measure the authors' interests.
The key difference between a harness and a leaderboard lies in the purpose. Benchmarks are crafted to demonstrate specific aspects of a model, while a harness like the one presented here helps address specific failure modes, such as model hallucination, inappropriate test writing, or SQL mangling. By running the same 20-task suite against multiple models, you can evaluate their performance and track any behavior drifts over time.
This process is not only cost-effective due to the use of free server options like MonkeyCode but also allows for frequent re-evaluation of model updates, ensuring that you have the most up-to-date insight into your model selection decisions.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
