Retries didn't make GitHub resilient. They DDoS'd it.

On August 17, 2026, GitHub experienced a seven-hour and forty-seven minute outage spanning from 1:28 PM to 9:15 PM UTC. During this period, users encountered issues with Git operations, Actions, Issues, Pull Requests, and Copilot. The Web and API traffic error rate reached approximately 20%, while archive and raw-content downloads had a roughly 50% error rate.

The root cause of the outage was an Istio sidecar proxy reaching its concurrency limit. However, the system's autoscaling mechanism monitored the host application's metrics instead of the sidecar's proxy saturation, leading to no new instances being deployed and four HAProxy nodes exceeding their capacity limits. This resulted in traffic overflow and further degradation of the gateway.

The situation worsened due to a hidden bug in the VS Code Copilot extension, which caused clients to excessively re-request authentication tokens from the Copilot Token Service. This led to backend errors and a retry loop triggered by the clients, rapidly increasing traffic during the recovery process. GitHub's CTO, Vlad Fedorov, explained that the retries were causing the problem to worsen, likening them to a self-denial of service attack.

The article emphasizes that retries, when not properly managed, can significantly exacerbate the issue. It highlights the importance of implementing budgeting and jitter in retry logic to prevent a surge in traffic and allow the system to recover. The article concludes by cautioning that relying solely on retries without proper limits and delays can lead to more harm than good, transforming the client into a well-intentioned adversary rather than a resilient one.