{
  "id": 15807847,
  "title": "A canary you keep is a canary you start trusting",
  "url": "https://urgent.news/2026/10/20/a-canary-you-keep-is-a-canary-you-start-trusting",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-10-20T22:02:46.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/whalejr/a-canary-you-keep-is-a-canary-you-start-trusting-1d81"
  },
  "original_language": "en",
  "account": "A reader, Zhang Enquan, revisited a piece I wrote about green builds. He analyzed 10,000 prompts, discovering 40.5% of samples deviated by more than 20% from the pipeline. The errors were not merely an outlier, but embedded within the data. This highlights how suites can lie without triggering alerts. Averaging, while a polite solution, often hides the tail issues. Zhang posed the question: what's the most verifiable check in a suite? His answer: a canary, a single known prompt whose token count can be manually checked. Re-computing it for each run takes mere seconds and detects when the pipeline inadvertently skips samples. Trust, not compute, is the costly component. The rule they both adhere to: \"A canary you keep is a canary you start trusting.\" Once a canary becomes a known-good fixture, it's a green light. Three checks now necessary on any green board: Was the number accurate before this release, merely copied in? Are the failures residing in the tail, not just the mean? Can the check be verified by hand within a minute? If not, Zhang doesn't consider it his check. This is not about tooling but rather whether anything in the room remains awake.",
  "summary": "A reader named Zhang Enquan came back to a thing I wrote about green builds. He had run 10,000 real prompts through a tokenizer, and the first thing he checked was not the output. It was whether every sample actually entered the pipeline. 40.5% of individual items were off by more than 20%. The batch looked about 5% off. The errors were not on top of the average. They were inside it, where nobody…",
  "key_points": [
    "40.5% of samples deviated by more than 20% from the pipeline",
    "A canary, a single known prompt, verifies pipeline integrity",
    "Trust in canary check is crucial for green builds"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}