Your Health Checks Are Verifying the Wrong Thing

Four bugs were discovered in one week, all of them passing monitoring checks. Each of these bugs was the same issue. The first bug involved a financial figure that was off by 16 times. A metric remained frozen for a month while appearing to update. There were nine decommissioned processes still holding live credentials. Additionally, there were 149 backup files that were serving secrets over HTTP to anyone who asked.

All of these issues passed the monitoring process without any problems. The problem lies in the fact that the monitoring checks were verifying the existence of something rather than its accuracy. The author claims that anyone who runs scheduled jobs, agents, or health reports has likely written this bug. The author provides an example of reporting monthly recurring revenue on an internal dashboard, where the number was wrong by a factor of sixteen for weeks, despite all monitoring checks passing.

The author emphasizes that reconciling against source data requires a second source, a join, and a tolerance, which was not done in this case. The author points out that a dashboard appearing green even when it is wrong is worse than one appearing red because green can create false confidence in a number that has not been checked.

The author then recounts three more instances where similar monitoring issues occurred, including a frozen metric, a weekly job that wrote the previous value forward due to an API failure, and decommissioned agents retaining access due to shared global secrets. In each case, the monitoring checks verified the existence of something rather than its truth.

The author concludes by outlining four rules that, if implemented, would have caught all of these issues. Every metric should carry its source and the time it was last genuinely verified, rather than the time it was last written. A value without provenance should render as unverified rather than as a number. The author stresses that green should mean reconciled against something independent, not just a formatting assertion.