Every Scheduled Task Was Refused for 70 Hours. My Watchdog Was One of Them.

On November 3 at 13:41, an error was logged for the GPU driver. This caused every new scheduled task to be refused, a problem that persisted for 70 hours. Despite the machine remaining on, no new tasks could start, including the watchdog. The ledger's last entry showed a date of November 2, and the visit count in reports remained unchanged, suggesting a seemingly quiet week. On November 6 at 11:57, the machine unexpectedly shut down and rebooted, allowing the scheduled tasks to function again.

The issue stemmed from the design of the autonomous agent, which relied on three scheduled tasks: one for collecting data, another for posting updates, and a third that monitored the system. All three tasks shared a common failure mechanism. The original monitor was a task that checked for updates in a log file every five minutes.

If the log did not update within 45 minutes, it sent a message over HTTP and repeated the check every three hours until the issue was resolved. However, this design had limitations. For instance, if the monitor process failed during the outage or before its initial start, it would not be able to recover. Additionally, if the entire machine stopped, the entire system would also be affected.

The cause of the initial state that led to the outage was unknown, as the scheduler's log was disabled.

To address these shortcomings, a new watchdog process was created. Unlike the previous setup, this watchdog stayed running once started and continuously monitored the log file. If no update was detected for more than 45 minutes, it sent an HTTP message and continued checking every three hours until the problem was fixed. This design change ensured that the watchdog would remain active even during outages, providing a more reliable detection method.

However, this solution still has gaps. The original monitor was still launched by a scheduled task, meaning it could fail if the task itself encountered issues or if the machine shut down unexpectedly. Furthermore, the exact cause of the initial error state remained unclear, as the system log was not available for review.