A green test is not a running reflex, and a running one is not a placed one

We run approximately 283 scheduled jobs on a few machines. Each job consists of a shell script that includes its own schedule, declares its own test, and gets automatically integrated into the cron system once the test passes. The configuration has a flaw that took five separate incidents to identify, as every incident appeared healthy from all perspectives.

Each of the following figures was measured on a 16-core Ubuntu 24.04 machine during the writing of this article, not sourced from the commit that resolved the issue. Two figures differed, and one mechanism failed to reproduce entirely. The core issue lies in the fact that "green" functions as a conjunction instead of a singular statement.

For a scheduled job to function correctly, four conditions must be met simultaneously: the job passes its test, the test confirms the task performed by the job, the job is indeed scheduled, and it is scheduled within a consumer's presence. We had monitoring in place for (1) and maintained a practice of verifying (2). However, we lacked any tools for (4), which silently operated for weeks without issue.

1. The detector examining the job's state

The first problem is minor in comparison to the difference and went unnoticed for six weeks in the production environment. A job fuses four inputs into a single node health label—HEALTHY, DEGRADED, CRITICAL—writes it to a state file, and with the --edge flag, prints a line only when the label changes. Cron executes this script every five minutes, while a separate log records the transitions.

The --edge function operates as follows: it writes the label to the state file, reads the previous state from the file, compares the previous state to the new state, and if they are the same, it exits without output. The previous state is read after the write, making it equal to the new state by construction. This equality test was consistently successful, and on every real transition, the --edge command exited with an empty output, preventing the transition log from appending entries.

The significance of this issue lies not in the ordering bug (which can be observed) but in the fact that every liveness signal we had indicated the job was operating normally, and each signal was accurate. All necessary components, such as the cron entry existing, the process running every five minutes, and the state file's modification time being current, were present.

The error was the absence of a read-before-write operation and the lack of serialization for the pair. The fix involves reading the state file before writing to it and serializing the pair. However, the second part of the fix, which I might have overlooked, is crucial: execute the pair of commands with the following Python code:

`python3 -c "a={m for m in range(2,60,5)}; b={m for m in range(2,60,15)} print(sorted(a & b))"`

This code outputs `[2, 17, 32, 47]`. The job is scheduled to run from 2-59/5. Another job—a vitality roll-up—has a different schedule of 2-59/15 and forces the refresh of the health label by calling the same tool. Consequently, every hour at minutes 2, 17, 32, and 47, two writers of the same state file attempt to access it simultaneously.

Without a lock, an interleaving that occurs at the exact moment of file truncation results in the reader receiving an empty `prev` value, which never matches a real label, causing a spurious edge. This issue was measured on the live file using the following commands: `$ grep -c ..` (33 lines naming both ends) and `$ grep -vc ..` (4 lines not naming both ends).

The four lines without "from" indicate the pre-fix survivors. These lines appear as follows: `[nodehealth-degraded] DEGRADED — sockstat=CONCERN No FROM side, no date`. The transition log lacks records of change or what changed, meaning it cannot attribute the events after the fact. Two failures compounded: the detector could not trigger, and the rare times it did trigger by accident, it produced unreadable entries.

Had we moved the read one line closer to the write instead of before it, the equality test would have become functional, and the race condition would have continuously generated those four lines. A partial fix addresses the log growth issue, providing the missing signal necessary to close the incident on the bug that was not resolved.