To Retry or Not to Retry: Safe Retry Logic with Exponential Backoff, Jitter and Idempotency Keys

In a previous incident, the team faced a notable issue where a payment provider took about 30 seconds to respond, and during that time, their system sent four requests to charge the same order. This happened due to three layers of retry: the client's SDK, the API service, and the job queue behind it. While no single layer was at fault, the outcome was disastrous.

From that experience, they learned that retry is not an added feature to ensure reliability. In fact, if it leads to incorrect behavior, it can be more dangerous than not retrying at all.

This article shares their approach to retry logic in production: when to retry, how to do it without overloading their system, and how to ensure safe retry for requests with side effects. The first question to ask before writing any code is whether the request should be retried. Before writing any code, they need to answer two questions: is this a temporary error (transient) or something else?

Does the 429 (Too Many Requests), 503 (Service Unavailable), 504 (Gateway Timeout), or 400/401/404/422 status code indicate a transient error? 400, 401, 404, 422 do not benefit from multiple retries. Is this request idempotent, or does it have an Idempotency-Key? HTTP GET, PUT, and DELETE methods are idempotent, while POST is not.

Retrying a POST /charges request that timed out is the quickest way to charge a customer twice, as timing out doesn't mean the server hasn't processed the request. There's a chance the server already processed it, and the response simply didn't reach the client.