Every Big Outage This Week Started With a Change
Email went down for a day. Every departing flight in the US was held for over four hours. A big bank's transactions stalled for most of a Monday. Social media decided it was a coordinated attack, a solar storm, or AI taking over. I read the official explanations, and they're far more boring than that. Almost every incident with a published cause traces back to a change: a code fix, a router swap,…
This week saw several major outages that all began with a single change. The most notable incidents included Microsoft 365 going down for a day due to a defect introduced in a July fix, the FAA ground stop lasting over four hours caused by a router installed during a fiber upgrade, Chase transactions halting for most of a day because of a software bug in an in-house hardware system, Google Cloud's London region experiencing about six hours of downtime due to a utility fault during planned maintenance, and social media initially suspecting coordinated attacks, solar storms, or AI takeover.
These incidents share commonalities such as the change and the failure being far apart in time, and the failure only manifesting under specific conditions. For instance, Microsoft's defect, introduced in July, only showed up under heavy load, newer high-core servers, and with about 14% of gateway capacity already out of service for unrelated reasons.
Similarly, Google's London zone experienced power loss while one of its power feeds was isolated for maintenance, which then led to more than 80% of gateways stopping response.
Recovery efforts often exacerbated the problems, as traffic was rerouted to healthy gateways which subsequently failed, creating a "thundering herd" situation where restarted servers were overwhelmed by reconnecting clients. The FAA's older equipment was particularly vulnerable when routers at two sites flooded the network with connection attempts, further complicating the diagnosis process due to some maintenance records being on paper.
While there is no definitive data proving a seasonal explanation, such as teams rushing changes before Thanksgiving code freeze, the timing of these incidents does coincide with the end of the year. However, it remains unclear whether the freeze itself has been effective or merely shifted risky deployments to the week before the freeze.
Tracking degraded but not down capacity, such as 14% of a fleet being offline, has also proven challenging, as these conditions can count as incidents before additional factors exacerbate the situation.
Defects can sit dormant for months before being triggered by the right load, as seen with the Microsoft defect and Google's London zone power loss. Those who manage changes at scale share varying experiences with change management practices, with some suggesting that a seasonal explanation may hold merit. The author, working at Control Plane, a cloud infrastructure company, invites readers who manage changes at scale to share their experiences and insights on what strategies have worked for them in preventing or mitigating such incidents.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.