A canary you keep is a canary you start trusting
A reader named Zhang Enquan came back to a thing I wrote about green builds. He had run 10,000 real prompts through a tokenizer, and the first thing he checked was not the output. It was whether every sample actually entered the pipeline. 40.5% of individual items were off by more than 20%. The batch looked about 5% off. The errors were not on top of the average. They were inside it, where nobody…
A reader, Zhang Enquan, revisited a piece I wrote about green builds. He analyzed 10,000 prompts, discovering 40.5% of samples deviated by more than 20% from the pipeline. The errors were not merely an outlier, but embedded within the data. This highlights how suites can lie without triggering alerts. Averaging, while a polite solution, often hides the tail issues.
Zhang posed the question: what's the most verifiable check in a suite? His answer: a canary, a single known prompt whose token count can be manually checked. Re-computing it for each run takes mere seconds and detects when the pipeline inadvertently skips samples. Trust, not compute, is the costly component. The rule they both adhere to: "A canary you keep is a canary you start trusting."
Once a canary becomes a known-good fixture, it's a green light. Three checks now necessary on any green board: Was the number accurate before this release, merely copied in? Are the failures residing in the tail, not just the mean? Can the check be verified by hand within a minute? If not, Zhang doesn't consider it his check. This is not about tooling but rather whether anything in the room remains awake.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
