Google engineer unplugged every fiber they could see and – surprise! – took down a chunk of the G-Cloud
This sounds like a RTFM error at hyperscale
A Google Cloud engineer caused a partial outage by unplugging every fiber-optic cable they could see during routine hardware maintenance in the us-central1-b region on September 1. This action led to "severe network degradation and resource isolation," with traffic flow drop rates reaching 100% and virtual machines becoming unreachable.
Google's service health report explained that the company's datacenters are built to be resilient against single-device and most multi-device failures. However, if all cables are unplugged, redundancy is defeated. The immediate cause was an engineer's procedural error that unplugged 100% of fiber paths across devices within 13 minutes.
Once the engineer disconnected the routers, virtual machines in one zone lost contact with the outside world while still being able to communicate with each other. Google's resilience provisions automatically shifted traffic to healthy capacity elsewhere in the region once the issue was noticed. Technicians reseated the disconnected optical links, restoring traffic flow rates and normalizing the traffic.
The incident highlights the importance of following safety procedures when maintaining infrastructure, as a simple oversight like unplugging cables can have significant consequences.
Written by urgent.news from The Register Science's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
This story
This is one outlet's version. Read the fullest account.
