Your pipeline deleted its own alarm (two greps to check)

Your daily report stopped arriving two days ago, unnoticed, because the absence of a message was indistinguishable from a quiet day. A reader suggested a guard that checks deployed canaries, running the job on production and verifying its identity, secrets, and result sink. This advice proved crucial, as it exposed a failure in the existing guard.

The guard initially appeared to work, but it failed on the second day, revealing an issue within the original probe. To address this, a second daily probe was implemented, which immediately detected the problem when run manually against production. The probe writes its history to a separate branch, allowing the main branch to remain clean.

However, the probe's mechanism can cause issues when checking out the branch, as it removes other files, including the notifier for the next step. This led to the daily message suddenly stopping, with only a red run as evidence. A minor change, passing on the day shipped and failing the next, can be the worst kind as it catches the team off guard.

By using only two greps and one command, teams can quickly identify scheduled workflows that push to a branch and check if any steps after the push run scripts from the repository. The final step is to manually trigger a scheduled job against production, observing its behavior, including any changes to permissions or notifiers. This approach provides real-time insights that no file in the repository could offer, helping prevent silent failures and ensuring the alarm still rings when necessary.