Retry Budgets and Dead-Letter Queues: Keeping Digital Delivery Reliable When Suppliers Fail

Retrying failed digital deliveries can quickly turn into a nightmare if not handled properly. In this post, the author shares a practical pattern that combines a retry budget with a dead-letter queue (DLQ) to keep digital delivery reliable even when suppliers fail.

Traditional retry implementations can exacerbate problems when a supplier is down. If multiple orders hit the same failing supplier simultaneously and each one retries several times, the number of requests against the struggling dependency multiplies, leading to a retry storm. This delays recovery, exhausts rate limits, and can cause other critical systems to become saturated.

A retry budget addresses this issue by capping retries globally within a rolling window, rather than retrying per order. For example, if 1,000 supplier calls are successful in a minute, a 20% retry budget would allow for at most 200 retries within that same window. Once the budget is exhausted, new failures bypass retries and go straight to a holding state, preventing unbounded retries from turning a brief supplier hiccup into a prolonged outage.

To implement this pattern, three components are required:

1. A counter tracking successful requests and retries, stored in Redis or in-memory with a rolling window.

2. A decision point in the delivery worker that checks the retry budget before scheduling a retry.

3. An escape hatch for manual replay when the supplier recovers.

The DLQ serves as a holding area for failed deliveries that have exhausted their retries or consumed the retry budget. Each DLQ entry contains full context, including the order ID, supplier, request payload hash, response code, timestamp, and retry history. The DLQ is not a place for orders to disappear; instead, it acts as a waiting area for deliberate decision-making.

The DLQ workflow has three paths:

1. Auto-replay: A background job attempts to replay DLQ items after a backoff, but only if the supplier is healthy. However, replays should not be scheduled blindly on a timer.

2. Supplier failover: If Supplier A continues to fail for a specific product, DLQ replays can be routed to an alternate supplier. This requires modeling substitutable inventory in the catalog.

3. Human review: If the code has already been issued but the email bounced, a human needs to resolve the state before issuing another code. Double-issuing is worse than a delay, as it creates duplicate codes or charges. Idempotency is crucial, using an idempotency key tied to the order and delivery attempt, and storing the result of the first successful fulfillment.

When implementing retries within the budget, certain practices matter:

* Use exponential backoff with jitter to avoid creating a thundering herd of retries at the same second. Set aggressive timeouts for supplier calls, as a quick failure is preferable to a worker being blocked for an extended period. Distinguish between retryable and non-retryable errors, such as distinguishing a 429 or 503 (retryable) from a 400 with an invalid product code (non-retryable).

* Log supplier error taxonomy, not just HTTP status codes. Different suppliers may return 500 for completely different reasons.

Observability is key to monitoring the reliability of digital delivery. Instead of alerting on error rates, focus on the depth and age of the DLQ. A growing DLQ indicates an issue before customers even notice. Track metrics such as delivery success rates by supplier and product, retry rates versus retry budget consumption, DLQ depth and oldest item age, replay success rates, and the time from order to first successful delivery (p50, p95, p99).

By implementing retry budgets and DLQs, digital delivery systems can gracefully handle supplier failures, turning a bad day into a handled case rather than a cascading outage. This approach turns the excuse "the supplier was down" into a managed and observable failure scenario. For storefronts selling digital codes, these reliability measures are essential, as customers care more about the timely arrival and functionality of the code than the underlying API response.

Regularly testing failure paths through chaos drills helps ensure the system behaves as expected when suppliers truly fail.