Your Scheduled Agent Says Success. It Did Nothing. Here's Why.

A green status from a scheduled agent run indicates that the process started and exited without any infrastructure errors. However, this status does not confirm that the task was completed successfully. An agent may fail to locate its input, misinterpret the repository it cloned, or simply decide there is no work to be done, yet it will still exit cleanly. To the scheduler, this appears identical to a run that performed the actual work.

In a three-brand automation fleet, three separate scheduled jobs reported green for days – one week in the worst case – yet produced no output. Neither the alerts nor the outside observers noticed any issues. To accurately define success, the artifact the job is supposed to write must be the defining criterion, not the exit code.

The solution is straightforward: every scheduled job must explicitly name the file it is supposed to produce. A successful run is only considered successful if it writes that file. Anything else – the job runs cleanly but produces nothing – should be recorded as a 'no_op' status, which is distinct from 'ok'. To monitor this without altering the job's internals, hash the target file before and after the run.

If the hash remains unchanged with a non-zero exit code, it signals a silent no-op, which your monitoring system can now detect.

One aspect that made the situation worse was a timestamp bug. In two of the jobs, the scheduled cron time was logged as the run time instead of the actual execution time. Since the staleness is computed from this field, a job that silently failed would still display a timestamp that appears fresh and healthy, giving a false impression of successful execution. Always record the wall-clock time from the moment of execution, not the intended schedule time.

Having a single supervisor that only checks whether the jobs that ran were successful is insufficient. It misses the jobs that stopped running entirely. True detection requires reconciliation in both directions: reading the declarations of every job that should exist and then diffing it against the run ledger. A declared job with no recent entry is stale, while an entry in the ledger that nothing declares is invisible work that goes unnoticed.

Caution is needed regarding false-alarm rates, as a high rate can cause real alerts to be ignored. During a trial run, the supervisor fired five alerts, but four of them were incorrect. They had miscounted a shared worker as three separate per-brand jobs, flagging the missing copies. The following day, a genuine outage produced an alert that mirrored the false alerts from the previous day, nearly leading to their dismissal.

To avoid this, monitor findings by confidence and only surface those that have been proven, even if it means holding back some alerts.

The critical check that ultimately led to changes was a weekly accuracy check that ran for two months. It generated reports every week, but two of those weeks resulted in fixes. The other weeks saw the same findings reappear because no action was required to address them. The remedy was to make the accuracy check gate a deploy, ensuring that a finding passes its grace period before a deployment occurs.

An alarm that is not required to be responded to is merely a suggestion, while an alarm that halts the pipeline is the only one that reliably triggers a fix.

In summary, to prevent scheduled agents from reporting success when they do nothing, implement the following five rules: an artifact-based success check, real execution timestamps, two-directional reconciliation between declared and actual runs, confidence-scored alerts, and at least one check that is wired to block something rather than simply report it.

The Distribution Autopilot Kit, which encapsulates these principles, was developed from the same fleet where these incidents occurred. However, these five rules are universally applicable and cost nothing.