Google lost a London zone, and the generators never started

At 07:00 London time on Tuesday 20 October 2026, one of Google Cloud’s three London zones, europe-west2-b, lost power. Servers, storage and network equipment in the data centre went dark, services across europe-west2 slowed or threw errors, and Google puts the main impact at 6 hours 8 minutes.

Nobody attacked it, and losing the mains was not the failure. The building is designed to survive that. The generators never started, because the controller that should have started them was checking for voltage in the wrong place, and both copies of it made the same check. Google’s own incident report, published on 27 October and revised on 4 November, sets it out in detail.

How the building is powered

The site runs on “a 2N utility power architecture”: in Google’s words, “the facility has complete duplicate power paths for every critical load.” Two independent utility feeds, A and B, supply it, with “a standby utility connection configured for manual campus-wide transfer during maintenance operations or emergency events.” Behind them sits “a central high-voltage generator bank managed by a dual-redundant Programmable Logic Controller (PLC).”

UPS batteries “provide approximately five minutes of autonomous runtime.” That is the window in which the PLC “detects utility undervoltage events, initiates diesel generator startup, synchronizes the generators, and transfers the site load.” Notice, start, transfer: all of it depends on the first step.

Before that morning

The site was not running in its normal shape. It had been placed in “a temporary single-feed (1N) bridged maintenance posture.” Feeder B’s breaker “was physically opened to allow for corrective repairs following issues identified during a quarterly generator load test on 14 Oct 2026,” and “remained in an open position” while the repairs went on. Feed A carried the whole critical load.

The report does not say what those issues were, and it does not say the test found the controller problem.

06:53: the generators never started

At 06:53 London time, the utility provider lost feed A, “caused by an external underground cable fault on the utility network outside the data center facility.” The batteries “sustained the critical load as intended.”

Google’s root cause: “a misconfiguration of the PLC logic that did not verify both an undervoltage condition and physical breaker position for both utility feeders. Instead, the PLC undervoltage sensing inputs were wired upstream (line side) of the isolated utility feed breakers.” The PLC “was programmed to verify whether voltage was present on the line side of the feeder before commanding a generator start.” Feed B was still energised upstream of its open breaker, so “the active and standby master PLCs, which were running identical control logic, sensed live voltage from feed B through shared upstream sensors on the disconnected path,” and “the PLC logic indicated that a healthy utility source was available … and did not issue a start command to the generator plant.”

The site team “attempted to execute a remote load transfer via the PLC; however, the system inhibited the action due to the live upstream voltage reading on feed B.” Recognising that the batteries would run out first, they moved the load to the standby utility source. The batteries were exhausted during that transfer: on Google’s timeline, at 07:00, seven minutes after the cable failed, zone b lost power.

The spread

The loss of zone b itself was, in Google’s words, “working within the design of a zonally-scoped system,” and zonal resources such as virtual machines “require customers to provision cross-zone failover and recovery mechanisms tailored to their needs.” Monitoring alerted Google’s engineers at 07:04, and at 07:09 traffic routing and client applications began failing over to the other London zones.

Two things spread the damage. First, “cross-zone remote procedure calls from regional service control planes directed to metadata and catalog components in the offline zone encountered timeouts,” stalling job submissions. Second, the surge of failover “saturated available compute slots and processing capacity in other availability zones,” with admission queues, delayed database failovers and intermittent 500 and 503 errors. Some customer failover attempts “were unsuccessful due to capacity constraints and difficulty in replicating the underlying storage mechanism.”

Google’s own summary of the regional effect: “Google Cloud designs regional services to be robust to zone failures, but in this case there were a set of unexpected failures which caused impact outside of the zonal failure domain.” One example from the report: Apigee X “is designed to withstand a single-zone loss”, but “a scheduling misconfiguration had placed two of three replicas in the zone that lost power.” The report lists 40 products affected in europe-west2, and found no impact outside the region.

Coming back

The standby supply was restored at 07:54 and power reached the data halls at 08:20. Clusters began recovering at 08:40 and network devices were back at 08:45. Storage took longer: with so many unclean shutdowns, “responders recognized this and chose not to restore services that would otherwise make this worse.” Traffic returned to the zone from 11:45, platform services and APIs were fully available at 12:11, and by 13:08 virtual machines were “99.5% operational and healthy.”

The first update on the incident’s status page is timestamped 09:35 London time, more than two and a half hours after the zone went dark, with “Workaround: None for now.” Google’s action list includes developing “an automated alert-to-communication mechanism for location-impacting incidents.”

What Google is changing

The controllers were reconfigured and validated “on both undervoltage and breaker status” (completed 26 October). Google also lists:

• a comprehensive PLC code review and resiliency testing against multiple failure scenarios, estimated in the 4 November revision for 6 November;
• generator transfer tests that “Test automatic startup of generators through PLC signals within the backup battery timer”, completed on 20 and 26 October;
• emergency procedures with “manual steps to attempt to restore active utility feed, manual restart of generators and manual transfer to other utility sources within the UPS battery run time”, practised twice per shift;
• validation of “global PLC configurations at sites with similar topologies and PLC manufacturers”, estimated for 15 November;
• PLC code reviews during commissioning, validated again by a second Google team within a quarter, estimated for 18 December;
• a freeze on planned maintenance at other sites with similar topologies, with elevated approvals for configuration changes outside standard designs, completed 30 October;
• faster notification and wider platform testing for infrastructure-level events.

Four lessons for infrastructure teams

A check that measures the wrong thing passes. The controller did what it was programmed to do: look for voltage. Google calls it “a voltage-sensing design flaw and PLC misconfiguration”: the logic did not check the breaker’s position as well as the voltage, and the sensing sat on the wrong side. Test where a check is looking, not only what it does with what it sees.

Redundancy does not cover a shared input. The site had two feeds and two controllers. With one feed open for repairs, both controllers, running identical logic on shared upstream sensors, gave the same wrong answer.

Maintenance is when hidden assumptions surface. Running on one feed was a temporary maintenance state, and it was the state that exposed this wiring. A degraded posture deserves its own test.

Zones that kept their power can still share a failure. Control-plane calls into a zone that has gone dark, and the capacity everyone plans to fail over into, are dependencies too.

Sources

• Google Cloud Service Health, incident Xs2XTHzAVC9YnafnPyzj, “Multiple services were experiencing issues in the europe-west2 region”: the Incident Report of 27 October 2026, revised on 4 November 2026, the Preliminary Incident Report of 22 October 2026, and the status updates of 20 October 2026. This article follows the revised report. Times here are London time (BST); the page gives US/Pacific. Where the preliminary and final reports differ (alert at 07:04 or 07:08, 6 hours 8 or 10 minutes), this article uses the final report.

Corrections are welcome, and any made are listed, dated, at the end of this article.