{
  "id": 3667538,
  "title": "Your AI Has a Reviewer. Has Anyone Ever Seen It Say No?",
  "url": "https://urgent.news/2026/08/27/your-ai-has-a-reviewer-has-anyone-ever-seen-it-say-no",
  "topic": "ai",
  "section": "AI",
  "published": "2026-08-27T04:25:16.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/heinrichneb/your-ai-has-a-reviewer-has-anyone-ever-seen-it-say-no-4ja8"
  },
  "original_language": "en",
  "account": "My AI has a reviewer, but has anyone ever seen it say no? The answer is yes, and a simple fix makes that clear. Two weeks ago, I counted 204 guards in my repositories, but 89% had never been shown they could fail. I fixed that for a batch, then filed a data-loss report about 1,000 files that were never lost, with no way to check it against a number.\n\nInitially, only about one in nine of the automated checks could prove they could fail. I addressed this by making a batch of guards prove they can fail, which was an overnight fix. However, it was not enough, and I discovered the embarrassing truth.\n\nEvery reviewer—whether a hand-written check, an LLM judge, or a second agent grading the first—needs a known-bad case wired through the live path. This unknown case is checked against the same entry point real work uses. The benchmark harness runs three gates per case, and if any gate fails, the case does not run. Unsolved states must go RED, and solved states must go GREEN. Known-bad cases must go RED again.\n\nWe take ten real recorded failures and have a model punch each one into a runnable check, creating ten runnable cases out of thirty gates. All thirty passed, zero discarded. The first attempt failed the internals filter because the model copied a product name from a comment, highlighting the importance of thorough testing.\n\nMost tools give you the audit log, not proof that the veto still fires. The reviewer I fixed can say no, but we need to see if it does. By adding a veto heartbeat—a date of the last refusal—visible and first-class, we make it clear when the veto has become stale. This simple addition provides a first glance at health and prevents silent failures.\n\nWhen I tried to check a long-running harvest job, I accidentally connected to the wrong machine, resulting in a thousand files being lost. However, there was nothing to contradict the false report because there was no dashboard, counter, or file with three numbers in it. The data was still sitting where it should be, and a second job was quietly running on it at that moment.",
  "summary": "Two weeks ago I counted 204 guards in my repos and found that 89 % had never been shown they can fail. I fixed that for a batch of them. Then tonight I filed a data-loss report about 1,000 files that were never lost — and nobody, including me, had a number to check it against. Last time I wrote here, the number was 204 and 11 %: of the automated checks in my repositories that draw a conclusion,…",
  "key_points": [
    "Only about one in nine automated checks could prove they could fail.",
    "Fixed batch of guards to prove they can fail overnight.",
    "Added veto heartbeat to show last refusal date."
  ],
  "editors_take": "Adding known-bad test cases and a veto heartbeat to AI reviewers ensures they can fail and provides transparency into their performance, preventing silent failures and promoting accountability.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}