Your cron job did not fail. It never ran.
There is a specific kind of outage that does not show up in any log you are reading. A backup job runs nightly at 3am. One night the machine reboots for a kernel patch and comes back with crond masked. No error is raised, because nothing ran. No alert fires, because nothing failed. The job's log file keeps its last entry from the night before, and that entry says SUCCESS . Eleven days later…
Many outages go unnoticed because they do not trigger any logs or alerts. A backup job, for example, runs nightly at 3am and logs a success message. If the machine reboots during the job, crond may be masked, resulting in no error or alert being raised. This absence of an event is not an event, and monitoring setups typically only watch for events, not their absence.
Checking exit codes and alerting on non-zero values is a common practice, but it is incomplete. A job can fail without producing an exit code due to various reasons, such as the crontab line having an unescaped % or the disk being full. These silent failures can be challenging to detect with standard monitoring rules.
To address this issue, a "dead man's switch" approach can be implemented. Instead of waiting for bad news, the job should report its success on a schedule. If the report does not arrive within the expected time frame, the lack of it becomes the alert. This can be achieved by adding two curl commands around the main job: one to check if the job has started, and another to confirm its completion.
The monitoring endpoint should never be able to fail the job itself, as this would turn a monitoring outage into a production outage.
When configuring the monitoring ping, it is crucial to report both the start and completion of the job. This provides information about the job's duration and helps catch long-running jobs that never produce an error. The job should also handle failures properly using a trap. The series value, generated by including the current timestamp and process ID in the curl request, ensures that the start and end pings are correctly correlated for each job run.
While monitoring the presence of job runs is essential, it is equally important to consider additional factors. Analyzing duration drift, comparing each run against a rolling median of the last 30 runs, can help identify increasing execution times. Overlap detection can also be valuable, as starting the next run while the previous one is still running may indicate a slow job or a data race.
However, it is crucial to set the appropriate grace period for absence alerts. Waiting for an exact one-minute gap between runs may result in false positives due to normal variations in job execution times. Instead, monitoring should be based on actual durations observed over a month, using the longest duration as the reference point. Alerts should be triggered after two consecutive missed runs for jobs that run more than once an hour, while a single missed run should be considered noise for daily jobs.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
