That Time Client Retries Turned a Recovery Into a 7-Hour Outage

In August, GitHub experienced a notable incident. A central component in the US encountered high load, leading to an extended recovery process. The root cause: clients repeatedly attempting to reconnect following a brief outage. As the system came back online, all clients that had been patiently waiting suddenly flooded it with retry requests at the same time, creating a so-called "retry-loop trap."

This phenomenon is common in various forms: health check endpoints returning 503 errors prompting clients to retry every second, connection pools overflowing and causing immediate reconnections, or a CDN origin going down with all edge nodes retrying at the same rate. The general flow is as follows: the system encounters a slowdown or failure, clients start retrying with a set interval, the system partially recovers, and then the retries flood the recovering system, causing it to go down again or remain slow, repeating the cycle until intervention occurs.

The recommended solution is exponential backoff with jitter. Instead of retrying every N seconds, the retries are spaced at increasing intervals with a random variation. For example, the retries could be at 1s, 2s, 4s, 8s, and 16s, with a +/- 20% variation. This approach helps spread out the retry load. Implementing circuit breakers also helps by entirely stopping the forwarding of requests to a failing service for a cooldown period.

The clients become aware that the circuit is open and fail fast instead of retrying. Server-side rate limiting can also be helpful, but during recovery, it's often better to gradually increase capacity rather than implementing a hard cutoff. Client-side rate limiting is frequently overlooked. Most HTTP clients will retry indefinitely by default.

The specific situation at GitHub was highlighted in their postmortem, which noted the record traffic that day - 115 million Actions runs and 2.9 billion monthly commits. This significant volume of automated systems retrying in a loop during recovery could lead to a "thundering herd" situation. The lesson here isn't that GitHub's infrastructure was faulty; rather, it's that a brief outage combined with automated clients employing naive retry logic can result in a thunderous outburst on recovery.

For services that clients depend on, it's crucial to consider the retry behavior of those clients. Are they utilizing exponential backoff? Do they have a circuit breaker in place? Or are they simply stuck in a loop with a 1-second interval? Similarly, if you are the client, it's essential to review your default retry configuration.

The default settings in many HTTP libraries are not conducive to helping recovering services. This incident serves as a valuable lesson for both service providers and clients.