Your AI Has a Reviewer. Has Anyone Ever Seen It Say No?
Two weeks ago I counted 204 guards in my repos and found that 89 % had never been shown they can fail. I fixed that for a batch of them. Then tonight I filed a data-loss report about 1,000 files that were never lost — and nobody, including me, had a number to check it against. Last time I wrote here, the number was 204 and 11 %: of the automated checks in my repositories that draw a conclusion,…
My AI has a reviewer, but has anyone ever seen it say no? The answer is yes, and a simple fix makes that clear. Two weeks ago, I counted 204 guards in my repositories, but 89% had never been shown they could fail. I fixed that for a batch, then filed a data-loss report about 1,000 files that were never lost, with no way to check it against a number.
Initially, only about one in nine of the automated checks could prove they could fail. I addressed this by making a batch of guards prove they can fail, which was an overnight fix. However, it was not enough, and I discovered the embarrassing truth.
Every reviewer—whether a hand-written check, an LLM judge, or a second agent grading the first—needs a known-bad case wired through the live path. This unknown case is checked against the same entry point real work uses. The benchmark harness runs three gates per case, and if any gate fails, the case does not run. Unsolved states must go RED, and solved states must go GREEN. Known-bad cases must go RED again.
We take ten real recorded failures and have a model punch each one into a runnable check, creating ten runnable cases out of thirty gates. All thirty passed, zero discarded. The first attempt failed the internals filter because the model copied a product name from a comment, highlighting the importance of thorough testing.
Most tools give you the audit log, not proof that the veto still fires. The reviewer I fixed can say no, but we need to see if it does. By adding a veto heartbeat—a date of the last refusal—visible and first-class, we make it clear when the veto has become stale. This simple addition provides a first glance at health and prevents silent failures.
When I tried to check a long-running harvest job, I accidentally connected to the wrong machine, resulting in a thousand files being lost. However, there was nothing to contradict the false report because there was no dashboard, counter, or file with three numbers in it. The data was still sitting where it should be, and a second job was quietly running on it at that moment.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
