The Budget Guard Refunded Twice, So the Batch Outlived Its Cap

The budget guard system refunded the same reservation on every retry, causing the batch to surpass its capacity limit and half of the shards to fail. The fix was to allow a reservation to be closed exactly once. The log shapes were rebuilt from a synthetic reproduction, not a real customer run, but the pattern was real. The numbers were illustrative, with the alert showing 78% and the provider indicating 100%.

The batch stopped at 04:10, with three of twelve shards failing due to quota errors. The local counter showed 78% of the budget consumed, while the provider dashboard revealed a full 100% usage. Both numbers stemmed from the same run, but only one was accurate. The question arose: which one should we trust when the tool we built is the one deceiving us?

The answer is clear – trust the ledger, not the counter. The initial approach was to stop counting and start logging. A single integer cannot be debugged, as it has no history or owner. Therefore, the counter was replaced with an append-only event log, where every state change writes a single line. The log contained three event types: reserve when a call starts, settle when real usage is known, and refund when nothing was consumed.

The reservation identity (rid) was crucial, as it was where the bug was hiding. The replay of the ledger proved more productive than rereading the code. The events were parsed and counted, revealing 412 reserves and 449 refunds. This discrepancy between refunds and reserves indicated the bug, as no accounting system can refund more often than it reserves.

This single line of output exposed the entire issue. Furthermore, only one of the two refund call sites was identified, revealing the root cause of the problem.