When datacenters lose power, generators should kick in. That didn’t happen for Google in London
October outage caused by bad PLC config exposed at a time maintenance work reduced resilience
On October 20, Google experienced a significant outage at its Europe-West-2-B datacenter in London. The root cause turned out to be a misconfiguration in the company's systems, which led to emergency generators failing to activate. This critical failure affected various businesses, including Lloyds, HSBC, Royal Mail, and Revolut.
Google disclosed the issue on its November 4 service status page, explaining that the facility relied on a 2N utility power architecture backed by a central high-voltage generator bank managed by a dual-redundant Programmable Logic Controller (PLC). The power supply was sourced from two independent utility feeders, A and B, with a standby utility connection for manual transfer during maintenance or emergencies.
However, a broken underground cable connected to primary feeder A caused a circuit breaker to open, leaving feeder B open and preventing the generators from receiving an automatic start signal. While Google's UPSes kept the facility running for five minutes, they eventually ran out of power, resulting in the outage. The search giant identified the PLCs as not being configured to handle the specific conditions, and promptly reprogrammed them.
Since then, Google has made several modifications to the electrical setup, conducted extensive testing, and implemented two action items: enhancing communication during location-impacting incidents and expanding platform-wide testing for infrastructure-level events. This experience likely prompts Google to improve its outage handling and customer communication for future events.
Written by urgent.news from The Register's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
