GitHub says its 7+ hour August 17 outage was caused by a capacity failure when peak traffic overwhelmed an infrastructure component in a Central US data center (Vlad Fedorov/The GitHub Blog)

On August 17, GitHub suffered a significant outage that lasted 7 hours and 47 minutes, disrupting various services including github.com, authentication, GitHub Actions, APIs, pull requests, issues, and Copilot. This outage affected developers and organizations globally, leaving them disappointed, as it marked the second major incident in August following an actions failure on August 6.

In March and April, the company had shared updates on the work being done to enhance GitHub's reliability. While progress has been made, these incidents underscore the need for accelerated improvements.

The investigation revealed that the outage commenced when traffic exceeded a new peak, causing a critical infrastructure component in the Central US data center to fail in scaling with the increased load. This capacity pressure then propagated through the systems, leading to authentication failures and disrupting multiple GitHub services.

Recovery involved rerouting traffic, isolating affected infrastructure, and restoring services in stages. Most services recovered earlier that day; however, certain Copilot services took longer, experiencing errors that triggered a client-side retry loop, thereby increasing traffic during the recovery process. To address this, the team had to mitigate the behavior before resuming normal traffic.

Neither outage was due to a code or configuration change; instead, they were capacity failures at their core. The failure to scale critical components before demand surpassed their capacity led to these incidents. Since April, monthly commits had surged from 1.4 billion to 2.9 billion, explaining the pressure on systems, yet this growth does not justify the outages.

GitHub had made earlier commitments to reliability, focusing on three priorities: adding capacity, improving efficiency, and removing architectural bottlenecks. These efforts have resulted in the addition of over 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity. Hardware was installed in existing data centers until additional power became a limiting factor, while the migration to Azure accelerated.

Azure now handles approximately 58% of GitHub's platform load and manages half of all Git operations, up from 12% in May. This expanded footprint has also aided the scaling of the largest monorepos. The next milestone is to achieve an architecture that supports linear read capacity growth with the number of readers, allowing unlimited read operations.

This will be rolled out gradually, starting with the largest monorepos. However, scale is not the only challenge; as the pace and complexity of changes have increased, existing operational practices have not kept pace. Teams and resources have been redirected toward enhancing availability, with investments made in stronger testing, safer rollouts, better observability, and more effective alerting. Despite these advancements, the work towards availability is not yet complete.

Additionally, critical systems are being isolated, and shared dependencies are being removed to reduce the likelihood and impact of future outages. Following the August 6 and August 17 incidents, two immediate changes were implemented. First, consistent retry limits, retry budgets, and variable timeouts were applied across service-to-service interactions to prevent retry storms and cascading load.

Second, lower-priority CPU and memory alerts were reviewed to identify components susceptible to failure during sudden traffic spikes. The company's commitment to high availability is not merely a technical promise; it is a responsibility, as the developer community depends on GitHub to build, ship, and operate their work. The August 17 outage demonstrated a failure to meet this responsibility, and it is the company's duty to rectify this.

Through scaling and reliability improvements, GitHub aims to rebuild trust and ensure a dependable platform for developers.