{
  "id": 2453433,
  "title": "We built a benchmark, then caught it strangling the models it was grading",
  "url": "https://urgent.news/2026/08/21/we-built-a-benchmark-then-caught-it-strangling-the-models-it-was",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-21T23:37:46.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/fortitudeomnis/we-built-a-benchmark-then-caught-it-strangling-the-models-it-was-grading-27gl"
  },
  "original_language": "en",
  "account": "The story begins with the creation of OmnisBench, an open benchmark for LLM routing, with a focus on LLM Router OmnisRouter. The author shares their concerns about the potential bias in the benchmark due to the fact that the models might have already seen the answers to HumanEval and GSM8K tasks. To address this, the author builds a fresh split of LiveCodeBench, keeping only the problems published after the models could have trained on them. They then re-grade the policies on the old and new tasks side by side. However, the first numbers were grim, and both the cheapest model and the expensive frontier model performed poorly on the hard problems. The author discovered that the reason behind this was that the cheap model's output budget was limited, causing it to run out of room and cut off its responses. They decided to fix the output budget, run the benchmark again, and published the responses to address the issue. After making these changes, the results showed that the cheap model's performance improved significantly, and routing earned more points on the fresh set, reaching 33 points higher than the cheap model. The author emphasizes the importance of publishing responses to ensure transparency and avoid misleading numbers.",
  "summary": "A couple of day ago I posted about OmnisBench, our open benchmark for LLM routing, specifically our LLM Router OmnisRouter , and made a fuss about how you can re-grade every number yourself because we publish the actual model responses. Two commenters, deanlee and jugeni, very politely pointed out that the whole thing might be resting on a fib. Their point: HumanEval and GSM8K are old. The models…",
  "key_points": [
    "OmnisBench created to benchmark LLM routing with focus on OmnisRouter",
    "Fresh split of LiveCodeBench excludes problems models could have trained on",
    "Cheap model's output budget limited, causing poor performance on hard problems"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}