The flag file my watchdog wrote was 0 bytes, so the alert fired for 8 days
A monitoring script is supposed to stop shouting once you've been told. Mine shouted every morning for eight days, and the reason was a file with zero bytes. The design that looked fine The pattern is common: a cron job runs every hour, checks something, and when the check fails it sends a message. To avoid a message every hour, it keeps a latch — a small JSON file with the date of the last time…
A monitoring script kept yelling daily for eight days, all due to a zero-byte flag file. The script was supposed to stop alerting once it had already notified of an issue. Instead, it kept a latch file with the date of the last warning. If the file already existed, the job remained silent. The issue occurred when the job tried to write to the latch file, but the time module was not imported.
This resulted in a NameError: name 'time' is not defined, which propagated to the caller and caused the script to exit. The existing, 0-byte latch file was mistakenly interpreted as "never warned," triggering the alert every day. The three main issues identified were 1) verifying that the latch file is in a proper state, 2) handling exceptions more effectively, and 3) trusting status fields blindly.
These problems stemmed from a proxy misrepresenting reality, with the alert being green due to a fake measurement. The author now emphasizes the importance of verifying each check with concrete evidence before assuming everything is working correctly.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
