48-Hour Field Notes: An NTP Step Ate 40 Seconds of My Batch Budget
Have you ever watched a job with a 20-minute budget give up at minute nineteen, on a host where every single call was fast? I spent two days chasing exactly that, and the fault was not in the network, the endpoint, or the model. My deadline accounting was reading the wall clock, and the wall clock on that box moved underneath me. These are my field notes from a 48-hour window: what I ran, what…
In the span of two days, a small batch job designed to classify failure lines from a log file was plagued by two issues that defied easy explanation. The first problem stemmed from using the time.time() function to measure the budget, which can be adjusted by NTP, hypervisors, or even manual date manipulations. This led to the job prematurely terminating, as the reported elapsed time was consistently 40 seconds longer than the actual elapsed time.
The second issue arose from a parser error that could not handle HTTP-dates in the Retry-After header, instead treating them as delta-seconds. This caused the job to repeatedly hammer the endpoint, despite receiving a polite "wait" message. To address these problems, the author implemented an injectable clock and an honest parser, allowing the failure to be tested on a laptop instead of a remote server at midnight.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
