{
  "id": 984908,
  "title": "Your pipeline deleted its own alarm (two greps to check)",
  "url": "https://urgent.news/2026/08/15/your-pipeline-deleted-its-own-alarm-two-greps-to-check",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-15T07:25:06.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/heinrichneb/your-pipeline-deleted-its-own-alarm-two-greps-to-check-3a1m"
  },
  "original_language": "en",
  "account": "Your daily report stopped arriving two days ago, unnoticed, because the absence of a message was indistinguishable from a quiet day. A reader suggested a guard that checks deployed canaries, running the job on production and verifying its identity, secrets, and result sink. This advice proved crucial, as it exposed a failure in the existing guard. The guard initially appeared to work, but it failed on the second day, revealing an issue within the original probe. To address this, a second daily probe was implemented, which immediately detected the problem when run manually against production. The probe writes its history to a separate branch, allowing the main branch to remain clean. However, the probe's mechanism can cause issues when checking out the branch, as it removes other files, including the notifier for the next step. This led to the daily message suddenly stopping, with only a red run as evidence. A minor change, passing on the day shipped and failing the next, can be the worst kind as it catches the team off guard. By using only two greps and one command, teams can quickly identify scheduled workflows that push to a branch and check if any steps after the push run scripts from the repository. The final step is to manually trigger a scheduled job against production, observing its behavior, including any changes to permissions or notifiers. This approach provides real-time insights that no file in the repository could offer, helping prevent silent failures and ensuring the alarm still rings when necessary.",
  "summary": "Silent failures Your daily report stopped arriving two days ago. Nobody noticed, because a message that does not arrive looks exactly like a quiet day. A reader told me my fix was only a promise Count the scheduled jobs in your repository that end by sending something. A report, a digest, a backup confirmation, an alert. Now say, without opening anything, which of them actually sent something…",
  "key_points": [
    "Guard initially appeared to work but failed on second day",
    "Second daily probe detected problem immediately",
    "Two greps and one command identify silent failures"
  ],
  "editors_take": "Implementing a guard that checks deployed canaries can help prevent silent failures by exposing issues within existing probes and ensuring alarms remain functional and effective.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}