A counter in process memory is not a guard: 131 restarts proved it

A critical oversight was discovered in a counter mechanism within a process memory, leading to 131 restarts of a daemon. The counter resided in a module-level variable and was reset to zero with every restart, preventing it from reaching a threshold of three. Consequently, despite failures, the process remained active, and no escalation was called.

The supervisor continued respawning the process on stale heartbeat, engaging in a crash loop without recognizing the loop as a failure. The ratio of escalations fired to daemon starts was zero over 1,501, indicating the guard's unreliability and potential irrelevance. To address this issue, a suggested design was introduced that eliminates counters in the alarm path and instead relies on timestamps.

By overwriting a one-line marker file with the current timestamp at the beginning of each round, the alarm can be triggered when the marker's age exceeds a specified threshold of seven days. This approach ensures that restarts do not interfere with the alarm's functionality, as timestamps are unaffected by restarts. Additionally, the design emphasizes the importance of the timestamp in determining the last time an event occurred, rather than counting the number of times a specific event has happened.

The reader highlights three distinct shapes of guards that never fired, including one where the guard was unreachable by construction due to a counter reset caused by restarts. The suggested fix involves moving the write operation to the top of every round, ensuring that even if a round skips the detector, the timestamp remains fresh and can still trigger the alarm.