Urgent.News

What's breaking now, across thousands of outlets.

Tech

Every Big Outage This Week Started With a Change

Email went down for a day. Every departing flight in the US was held for over four hours. A big bank's transactions stalled for most of a Monday. Social media decided it was a coordinated attack, a solar storm, or AI taking over. I read the official explanations, and they're far more boring than that. Almost every incident with a published cause traces back to a change: a code fix, a router swap,…

This week saw several major outages that all began with a single change. The most notable incidents included Microsoft 365 going down for a day due to a defect introduced in a July fix, the FAA ground stop lasting over four hours caused by a router installed during a fiber upgrade, Chase transactions halting for most of a day because of a software bug in an in-house hardware system, Google Cloud's London region experiencing about six hours of downtime due to a utility fault during planned maintenance, and social media initially suspecting coordinated attacks, solar storms, or AI takeover.

These incidents share commonalities such as the change and the failure being far apart in time, and the failure only manifesting under specific conditions. For instance, Microsoft's defect, introduced in July, only showed up under heavy load, newer high-core servers, and with about 14% of gateway capacity already out of service for unrelated reasons.

Similarly, Google's London zone experienced power loss while one of its power feeds was isolated for maintenance, which then led to more than 80% of gateways stopping response.

Recovery efforts often exacerbated the problems, as traffic was rerouted to healthy gateways which subsequently failed, creating a "thundering herd" situation where restarted servers were overwhelmed by reconnecting clients. The FAA's older equipment was particularly vulnerable when routers at two sites flooded the network with connection attempts, further complicating the diagnosis process due to some maintenance records being on paper.

While there is no definitive data proving a seasonal explanation, such as teams rushing changes before Thanksgiving code freeze, the timing of these incidents does coincide with the end of the year. However, it remains unclear whether the freeze itself has been effective or merely shifted risky deployments to the week before the freeze.

Tracking degraded but not down capacity, such as 14% of a fleet being offline, has also proven challenging, as these conditions can count as incidents before additional factors exacerbate the situation.

Defects can sit dormant for months before being triggered by the right load, as seen with the Microsoft defect and Google's London zone power loss. Those who manage changes at scale share varying experiences with change management practices, with some suggesting that a seasonal explanation may hold merit. The author, working at Control Plane, a cloud infrastructure company, invites readers who manage changes at scale to share their experiences and insights on what strategies have worked for them in preventing or mitigating such incidents.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

I almost unlocked Pro from a URL parameter. What I changed before launch.

I build a small SaaS called ClauseRadar ( https://clauseradar.online/?utm_source=devto&utm_medium=article&utm_campaign=paywall-lessons ).

  • Replaced CSS blur preview with AES-256-GCM encryption and unlock endpoint for secure Pro status
  • Fixed installer bug by reading mount target from main.js and implementing fail-open loading state

The Importance of Documentation in Software Development

Documentation is sometimes treated as an optional part of software development. Developers often focus on implementation, features, and fixing bugs, leaving documentation until later.

  • Documentation often overlooked, developers focus on coding
  • Clear introduction and README vital for initial understanding
  • Up-to-date, navigable documentation enhances team collaboration

Design a 3D Printed Coaster

AI disclosure: This edition was prepared by an AI agent from the published Reactor 3D tutorial and its original interface screenshots.

  • 3D printed coaster designed using Reactor software
  • Coaster features rounded square outline and cup recess
  • Seven channels added to collect condensation

More from Tuesday 17 November →