{
  "id": 21171798,
  "title": "I spent a day breaking my own eval tool's scoring rule, and it was the best use of the day",
  "url": "https://urgent.news/2026/11/22/i-spent-a-day-breaking-my-own-eval-tools-scoring-rule-and-it-was-the",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-22T05:40:21.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/the_movie_9b1357e8c99cfaa/i-spent-a-day-breaking-my-own-eval-tools-scoring-rule-and-it-was-the-best-use-of-the-day-eak"
  },
  "original_language": "en",
  "account": "This reporter spent a day testing their own evaluation tool's scoring rule, and found it to be a valuable use of time. The evaluation tool, called AIEval, is a simple command-line interface that compares recorded model answers to expected strings and exits with a non-zero code if accuracy falls below a certain threshold. The reporter built a set of tests with a 0.80 accuracy threshold and fed it answers that should not pass. To their surprise, three out of four verdicts came from substring behavior, not the model being right or wrong. This highlights the trap of measuring accuracy based on short tokens, as the accuracy number can be partially a coincidence. To avoid this, the reporter suggests using expected strings that a wrong answer cannot accidentally contain. The reporter also discovered that a missing response is considered a failure, not a skip, and that max_avg_latency_ms is inert and not enforced by the harness. The evaluation process can be simplified to a single non-zero exit code in a CI step, making it easier to detect model regressions.",
  "summary": "This is the developer-facing version of a post I published on AI Stack Lab: Your Eval Suite Is Theater Until a Non-Zero Exit Code Stops the Merge . Same measurements, different angle — this one is about the scoring rule itself. I spent a day breaking my own eval tool's scoring rule, and it was the best use of the day Last week I shipped a $1 CLI called AIEval that does exactly one thing: read a…",
  "key_points": [
    "Reporter spent a day testing AIEval tool's scoring rule",
    "Three out of four verdicts based on substring behavior",
    "Suggests using expected strings to avoid accidental accuracy"
  ],
  "editors_take": "Relying on substring matching for accuracy metrics creates a false sense of security that can mask model failures and lead to unreliable evaluation results in automated testing pipelines.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}