The HTTP 429 That Turned Seven Minutes Into Zero Work

In a recent incident during DEV's Summer Bug Smash, a data-catalog governance system called ARGUS experienced a seven-minute outage due to an HTTP 429 error. This error code indicated that the user had exceeded a rate limit, either through a short burst limit or a daily token quota. The ARGUS system used structured model calls to draft descriptions, classify sensitive fields, and review repairs before any could be written. During a 45-entity sweep, the system spent seven minutes producing zero proposals.

The health probes returned a 200 status, indicating that the endpoint was reachable and the logs showed only a 429 Too Many Requests error. The retry loop was working as written, amplifying the outage. ARGUS treated every 429 as temporary and waited through six backoffs for each model call. A sweep could require roughly one hundred calls. For a rate limit that clears in seconds, this behavior was patient, but for a daily quota that resets tomorrow, it turned one deterministic failure into hours of waiting.

During the incident, ARGUS discarded the response body in the error boundary, which contained crucial information needed to fix the failure. The provider's daily-quota response explained the limit of 100,000 tokens, with 99,299 used and 1,139 requested. The ARGUS system never logged this information, making it difficult to identify the root cause of the outage.

To address these issues, the following fixes were implemented:

1. Preserve the provider's explanation: The HTTP boundary now carries a bounded copy of the response body when raising an error, including details such as which limit was hit, how much was used, and what the failed request needed.

2. Model exhaustion as its own outcome: A BudgetExhausted exception was added to fail the call immediately when the response identifies a long-lived token or request quota. This prevents the remaining agents from repeating the same doomed call and resets the breaker at the start of the next sweep.

3. Do not turn an outage into a verdict: A not reviewed outcome was introduced when the model budget is exhausted. This outcome is distinct from not approved or rejected, ensuring that the review gate is not silently removed at the exact moment it stops working.

4. Pace and rotate: ARGUS now uses header information to pace before hitting the rate limit and separates retry and rotation loops. This prevents the entire provider chain from being burned out by one bad prompt.

Additionally, the reset header parsing was updated to handle various duration formats, such as hours, minutes, seconds, milliseconds, decimals, and compound durations. Before this fix, the parser only handled simple float values, leading to ambiguous fallback risks and inaccurate reset durations.