{
  "id": 4427405,
  "title": "My test report printed \"0/96, 0% pass rate\". The truth was my account was out of credit",
  "url": "https://urgent.news/2026/08/30/my-test-report-printed-0-96-0-pass-rate-the-truth-was-my-account-was",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-30T13:05:13.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/dexterlung/my-test-report-printed-096-0-pass-rate-the-truth-was-my-account-was-out-of-credit-50ee"
  },
  "original_language": "en",
  "account": "A script that prints a score—such as an evaluation, CI check, audit tool, or health check—must be scrutinized to see what it prints if infrastructure fails. If the answer is a score, ratio, or pass count, the script is a false red light generator. The author discovered this after developing with Claude Code and accumulating 89 custom skills, each with a self-description that gets packed into the model's opening memory. When the character limit was exceeded, older descriptions were truncated, potentially causing the model to stop invoking the skill without any error. The author wrote a reconciliation script to compare old and new descriptions, restoring 8 skills that had lost trigger words. However, the author initially believed that a question bank of 4 cases per skill did not exist, as it was mentioned in a handoff document. But after running a full audit to push changes, the author noticed that one check had gone from \"pass\" to \"blind\" and took 100 seconds, indicating that the question bank did exist. This led the author to realize that the script that printed the score was already displaying a false red light before their changes. The author added a new convention to classify each exception as either infrastructure being down or the skill genuinely not triggering. Credit, key, rate limit, overload, DNS, and network issues were categorized as \"blind,\" which does not count towards the pass rate. The per-case \"✗ passed 0/3\" detail was removed for blind cases, preventing the false red light from appearing. The author also added reverse tests to ensure that a real failure is not classified as blind, which would mask a real issue as a false positive. After implementing these changes, the author verified that the audit output no longer displayed the false red light, showing \"no pass rate\" instead of \"0/96 (0%).\"",
  "summary": "Something you can do right now Open any script you have that prints a score — an eval, a CI check, an audit tool, a health check — and ask it one question: If the infrastructure fails (out of credit, expired key, rate limited, no network), what does this script print? If the answer is a score, a ratio, or a pass count , you own a false-red-light generator. Not \"might go wrong.\" Structurally…",
  "key_points": [
    "Script printed false red light due to infrastructure failure",
    "Author discovered missing question bank of 4 cases per skill",
    "False red light removed after adding blind case classification"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}