Sent is not confirmed
A send tells you about your side. A read-back tells you about the other side. Most agent failures I have debugged this year live in the gap between those two sentences, in a state that deserves its own name: sent, unconfirmed. Three times it bit us, in three different costumes. The timers that were not running. A fleet of scheduled jobs showed as loaded on a dashboard. Every run was dying…
Reports from the source reveal a significant issue in the system where a "sent" state does not necessarily confirm the subsequent "running" state. This "sent, unconfirmed" state, which the author argues deserves its own name, was observed multiple times in different scenarios. For instance, timers that were not running, loaded jobs showing as such on dashboards, and missing libraries affecting job execution were all examples of this phenomenon.
The root cause of this issue was a discrepancy between the dashboard displaying the schedule and the actual work being performed. The solution was not to improve the dashboard, but to rely on the run log, which contained the truth. The author emphasizes that the presence of a fresh file from the wrong job does not guarantee the right thing ran, and a status check must verify the expected artifact's hash to confirm the process.
This "sent, unconfirmed" state can lead to incorrect conclusions due to the oversight of an old backup script writing into the same folder. Furthermore, the author highlights that a request's confirmation is often delayed, causing the system to round up to a "done" state prematurely. This can result in unresolved requests being recorded as paid, even if they were never finalized.
To address this, the author suggests adhering to a set of rules: only write and then read back from a different surface than the one that sent the request, and only then decide. Retrying should be a last resort, and only when a double-apply is harmless. The author also stresses the importance of naming the unconfirmed state in reports to maintain honesty, as the term "done" does not accurately reflect the situation when the other end cannot be confirmed.
The final piece of advice is to give unconfirmed a deadline, after which a failure to confirm should be recorded rather than maintaining an unverified state. The author concludes that these practices, which have been in place for decades, provide a clear distinction between write and fsync operations in filesystems, and advocate for a disciplined approach in reporting that focuses on the climate rather than the weather.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
