Our watchdog restarted a healthy agent 24 times a day for months, and its own log said everything was fine

The watchdog in question restarted a healthy agent 24 times a day for several months. Initially, it appeared as though the watchdog was effectively saving the agent each time it restarted, leading to an inaccurate conclusion. However, upon closer examination, it was found that the watchdog was incorrectly detecting restarts based on log lines, rather than actual process crashes.

The intervals between restarts were measured, revealing a consistent pattern of approximately every hour, which was identified through process age measurements rather than relying solely on the log.

The watchdog's health check mechanism was discovered to be flawed. It determined if the agent was "hung" by checking for a specific tag in the log within a 30-minute window. However, this method proved inadequate as idle long polling could go unnoticed for extended periods without generating log lines. Consequently, a healthy process would be mistakenly identified as hung and subsequently restarted. This false sense of vigilance, while documented in the log, failed to accurately reflect the agent's true state.

The article emphasizes the importance of measuring the actual process health rather than relying on the agent's self-assessment. The provided "heartbeat" from the work suggests that a more reliable approach would be for the agent to write a file indicating successful completion of a task, allowing the watchdog to restart only when that file goes missing or becomes stale. This change would prevent idleness from being confused with a process crash, thereby avoiding unnecessary restarts.

Furthermore, the article explores the behavior of different frameworks when encountering various issues. LangGraph, OpenAI Agents SDK, and CrewAI were tested under conditions of flaky endpoints, tools that consistently fail, and tools that return invalid JSON. The results showed that while each framework exhibited different behaviors, CrewAI emerged as the most resilient, thanks to its underlying retry mechanism in litellm.

In contrast, LangGraph and OpenAI Agents SDK exhibited less robustness, with LangGraph stopping after a single model call and OpenAI Agents SDK making multiple calls in an attempt to resolve the issue.

The article concludes by highlighting the significance of measuring process health and implementing appropriate heartbeat mechanisms. It stresses the need to write a heartbeat from the work itself, ensuring that the agent is restarted only when there is a genuine issue, rather than as a result of a flawed health check system. The provided MIT-licensed code, along with instructions on how to run it, is offered as a practical solution for assessing agent uptime without the need for any paid keys.