A scheduled job can be healthy while its work is overdue

A scheduled job can appear healthy on the surface while its actual work remains overdue, as demonstrated by a small local-AI news pipeline running on a Mac. Despite the system showing that three jobs - news drafting, model checks, and reports - all registered successfully without any recorded consecutive failures, the last successful news generation was older than its scheduled interval.

The dashboard falsely answered the question "does the job exist?" when the real question should have been "has the expected work completed?"

The pipeline checked for overdue work every five minutes, with news drafting due every six hours and model checks and reports weekly. A shared execution lock was employed to prevent overlapping work among the three jobs. On September 24, a check at 18:20 Korea time still displayed the previous day's successful news run. The scheduler logs revealed repeated "busy" results, but the earlier entries lacked timestamps.

A power log analysis revealed sleep and wake events, yet there was no indication of a permanently stuck process.

The following day, a scheduled run commenced at 18:32 and successfully completed at 18:34, before the implementation of a locking change. Three source items were selected, and English and Korean drafts were generated. However, two automated quality warnings persisted, and those drafts remained unpublished. The revised lock acquisition order was: first, verify whether the task was due (utilizing state read), and then acquire a shared lock with bounded retries.

If the task was not due, the process would return "not_due". The second state check ensures that even if another process completes the work after the initial check but before the lock is acquired, it will not be executed.

The status check was modified to differentiate three crucial factors: whether the operating system has registered the job, when the scheduler last checked and the reason behind the run or skip, and when the useful work last completed, along with what output was produced. Any due task lacking a completed run is highlighted rather than being concealed behind an "zero failures" message.

A skipped run would now be made visible, with a warning provided for a due task without a completed run, as well as for a missing recent check, attributing possible explanations such as sleep and logout, rather than immediately diagnosing crashes. Regression tests covered scenarios like a not-due job avoiding the lock, a brief collision followed by execution, and a prolonged collision without recording success, resulting in a 40-test pass at a specific checkpoint.

While this diagnosis provides insights based on the available records, it does not establish a permanent solution for unattended operation. The Mac's ability to sleep is still preserved, with battery usage not impeding scheduled work while the device is awake. The local model is unloaded after a brief idle period instead of being kept resident between distant jobs, a decision documented in Ollama's model retention settings.

Ultimately, the practical lesson extends beyond the notion of "locks being bad." It emphasizes the importance of checking if a task contains work before making it compete for an execution lock, rechecking upon acquiring the lock, and honestly reporting skipped work. Registration, execution, generation, and publication are distinct milestones, and a comprehensive status screen should not amalgamate them into a single light.