Four Identical Red Runs: One Was Our Watchdog Working, Three Were Its Corpse
A cron in one of our repositories stopped firing on 21 June and nobody noticed for ten weeks. Its only job was to rebuild this site each morning so that blog posts whose publish date had arrived would actually appear. So for ten weeks, scheduled posts did not publish. The failure mode of a cron is silence, and silence is the one thing no dashboard renders. We built a watchdog for it. The watchdog…
On 21 June, a cron in one of the repositories stopped working, but nobody noticed until ten weeks later. The cron's sole job was to rebuild the site each morning so that blog posts with their scheduled publish date would appear. A watchdog was built to monitor the cron, which had four scheduled runs and all four resulted in red, indicating failure.
The first red was the watchdog working as intended, while the other three were the result of the watchdog crashing before it could check anything. The four runs exhibited the same failure pattern, with step 5 checking every scheduled workflow and step 6 filing an issue when a schedule has stopped. When the watchdog found a stopped cron, it filed an issue about it on 30 August.
However, it crashed in step 5 on 31 August, 1 September, and 2 September, with no further reports. The issue was not due to a commit, but rather the activation of Dependabot, which created an extra workflow. The watchdog, unaware of this new workflow, continued to collect data, leading to its own failure. The problem was a design flaw, as every outcome of the watchdog's reporting step collapsed into one signal, resulting in all failures being reported as red.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
