# Google lost a London zone, and the generators never started > A Google Cloud zone in London went dark on 20 October 2026 after its generator controller read voltage on the wrong side of an open breaker. - URL: https://trustboundarystudio.com/posts/google-cloud-london-power-2026/ - Incident: Google Cloud, October 2026 - Published: 2026-11-05 - Video: https://youtu.be/D3OYDRnlzSo - Topics: Google Cloud, europe-west2, post-mortem, data centre power, availability zones, failover, SRE, incident response ## Key facts - **When:** Tuesday 20 October 2026. A utility cable fault cut feed A at 06:53 London time; zone europe-west2-b lost power at 07:00. Google puts the main impact at 6 hours 8 minutes. - **Root cause:** In Google's words, "a misconfiguration of the PLC logic that did not verify both an undervoltage condition and physical breaker position for both utility feeders." The sensing inputs were wired upstream of the isolated feed breakers, so both controllers, reading shared sensors, saw live voltage and issued no generator start. - **Why feed B was out:** The site was in "a temporary single-feed (1N) bridged maintenance posture": feeder B's breaker was physically opened for repairs after issues found in a quarterly generator load test on 14 October 2026. - **Beyond zone b:** Some workloads in zones europe-west2-a and -c were degraded by demand surges and control-plane dependencies across zonal boundaries. Google: "there were a set of unexpected failures which caused impact outside of the zonal failure domain." Its report lists 40 affected products and found no impact outside europe-west2. - **Fix:** The controllers were reconfigured and validated on both undervoltage and breaker status (completed 26 October). Google set out to validate controller configurations at similar sites (estimated 15 November), and froze planned maintenance at other sites with similar topologies (completed 30 October). - **Primary source:** Google Cloud Service Health, incident Xs2XTHzAVC9YnafnPyzj: the Incident Report of 27 October 2026, revised on 4 November, the Preliminary Incident Report of 22 October, and the status updates of 20 October. Google names no affected customers. At 07:00 London time on Tuesday 20 October 2026, one of Google Cloud's three London zones, europe-west2-b, lost power. Servers, storage and network equipment in the data centre went dark, services across europe-west2 slowed or threw errors, and Google puts the main impact at 6 hours 8 minutes. Nobody attacked it, and losing the mains was not the failure. The building is designed to survive that. The generators never started, because the controller that should have started them was checking for voltage in the wrong place, and both copies of it made the same check. Google's own incident report, published on 27 October and revised on 4 November, sets it out in detail. ## How the building is powered The site runs on "a 2N utility power architecture": in Google's words, "the facility has complete duplicate power paths for every critical load." Two independent utility feeds, A and B, supply it, with "a standby utility connection configured for manual campus-wide transfer during maintenance operations or emergency events." Behind them sits "a central high-voltage generator bank managed by a dual-redundant Programmable Logic Controller (PLC)." UPS batteries "provide approximately five minutes of autonomous runtime." That is the window in which the PLC "detects utility undervoltage events, initiates diesel generator startup, synchronizes the generators, and transfers the site load." Notice, start, transfer: all of it depends on the first step. ## Before that morning The site was not running in its normal shape. It had been placed in "a temporary single-feed (1N) bridged maintenance posture." Feeder B's breaker "was physically opened to allow for corrective repairs following issues identified during a quarterly generator load test on 14 Oct 2026," and "remained in an open position" while the repairs went on. Feed A carried the whole critical load. The report does not say what those issues were, and it does not say the test found the controller problem. ## 06:53: the generators never started At 06:53 London time, the utility provider lost feed A, "caused by an external underground cable fault on the utility network outside the data center facility." The batteries "sustained the critical load as intended." Google's root cause: "a misconfiguration of the PLC logic that did not verify both an undervoltage condition and physical breaker position for both utility feeders. Instead, the PLC undervoltage sensing inputs were wired upstream (line side) of the isolated utility feed breakers." The PLC "was programmed to verify whether voltage was present on the line side of the feeder before commanding a generator start." Feed B was still energised upstream of its open breaker, so "the active and standby master PLCs, which were running identical control logic, sensed live voltage from feed B through shared upstream sensors on the disconnected path," and "the PLC logic indicated that a healthy utility source was available ... and did not issue a start command to the generator plant." The site team "attempted to execute a remote load transfer via the PLC; however, the system inhibited the action due to the live upstream voltage reading on feed B." Recognising that the batteries would run out first, they moved the load to the standby utility source. The batteries were exhausted during that transfer: on Google's timeline, at 07:00, seven minutes after the cable failed, zone b lost power. ## The spread The loss of zone b itself was, in Google's words, "working within the design of a zonally-scoped system," and zonal resources such as virtual machines "require customers to provision cross-zone failover and recovery mechanisms tailored to their needs." Monitoring alerted Google's engineers at 07:04, and at 07:09 traffic routing and client applications began failing over to the other London zones. Two things spread the damage. First, "cross-zone remote procedure calls from regional service control planes directed to metadata and catalog components in the offline zone encountered timeouts," stalling job submissions. Second, the surge of failover "saturated available compute slots and processing capacity in other availability zones," with admission queues, delayed database failovers and intermittent 500 and 503 errors. Some customer failover attempts "were unsuccessful due to capacity constraints and difficulty in replicating the underlying storage mechanism." Google's own summary of the regional effect: "Google Cloud designs regional services to be robust to zone failures, but in this case there were a set of unexpected failures which caused impact outside of the zonal failure domain." One example from the report: Apigee X "is designed to withstand a single-zone loss", but "a scheduling misconfiguration had placed two of three replicas in the zone that lost power." The report lists 40 products affected in europe-west2, and found no impact outside the region. ## Coming back The standby supply was restored at 07:54 and power reached the data halls at 08:20. Clusters began recovering at 08:40 and network devices were back at 08:45. Storage took longer: with so many unclean shutdowns, "responders recognized this and chose not to restore services that would otherwise make this worse." Traffic returned to the zone from 11:45, platform services and APIs were fully available at 12:11, and by 13:08 virtual machines were "99.5% operational and healthy." The first update on the incident's status page is timestamped 09:35 London time, more than two and a half hours after the zone went dark, with "Workaround: None for now." Google's action list includes developing "an automated alert-to-communication mechanism for location-impacting incidents." ## What Google is changing The controllers were reconfigured and validated "on both undervoltage and breaker status" (completed 26 October). Google also lists: - a comprehensive PLC code review and resiliency testing against multiple failure scenarios, estimated in the 4 November revision for 6 November; - generator transfer tests that "Test automatic startup of generators through PLC signals within the backup battery timer", completed on 20 and 26 October; - emergency procedures with "manual steps to attempt to restore active utility feed, manual restart of generators and manual transfer to other utility sources within the UPS battery run time", practised twice per shift; - validation of "global PLC configurations at sites with similar topologies and PLC manufacturers", estimated for 15 November; - PLC code reviews during commissioning, validated again by a second Google team within a quarter, estimated for 18 December; - a freeze on planned maintenance at other sites with similar topologies, with elevated approvals for configuration changes outside standard designs, completed 30 October; - faster notification and wider platform testing for infrastructure-level events. ## Four lessons for infrastructure teams **A check that measures the wrong thing passes.** The controller did what it was programmed to do: look for voltage. Google calls it "a voltage-sensing design flaw and PLC misconfiguration": the logic did not check the breaker's position as well as the voltage, and the sensing sat on the wrong side. Test where a check is looking, not only what it does with what it sees. **Redundancy does not cover a shared input.** The site had two feeds and two controllers. With one feed open for repairs, both controllers, running identical logic on shared upstream sensors, gave the same wrong answer. **Maintenance is when hidden assumptions surface.** Running on one feed was a temporary maintenance state, and it was the state that exposed this wiring. A degraded posture deserves its own test. **Zones that kept their power can still share a failure.** Control-plane calls into a zone that has gone dark, and the capacity everyone plans to fail over into, are dependencies too. ## Sources - Google Cloud Service Health, [incident Xs2XTHzAVC9YnafnPyzj](https://status.cloud.google.com/incidents/Xs2XTHzAVC9YnafnPyzj), "Multiple services were experiencing issues in the europe-west2 region": the Incident Report of 27 October 2026, revised on 4 November 2026, the Preliminary Incident Report of 22 October 2026, and the status updates of 20 October 2026. This article follows the revised report. Times here are London time (BST); the page gives US/Pacific. Where the preliminary and final reports differ (alert at 07:04 or 07:08, 6 hours 8 or 10 minutes), this article uses the final report. Corrections are welcome, and any made are listed, dated, at the end of this article. ## Questions this answers **What caused the Google Cloud europe-west2 outage on 20 October 2026?** An underground utility cable fault cut the only live feed to the data centre hosting europe-west2-b, while the other feed's breaker was open for repairs. The generators never received a start signal, because the controllers' voltage sensing sat on the live side of the open breaker, and the UPS batteries ran out. **Did Google's backup generators fail in London?** No. Google says "a voltage-sensing design flaw and PLC misconfiguration prevented the facility's generator plant from receiving an automatic start signal". Both programmable logic controllers concluded a healthy utility source was available and did not issue a start command. The batteries sustained the load as intended until they were used up. **Was all of Google Cloud London down?** No. Zone europe-west2-b lost power. Some workloads in zones a and c and regional services saw latency, errors and failed failovers from demand surges and control-plane dependencies, which Google describes as "a set of unexpected failures which caused impact outside of the zonal failure domain." **How long did the Google Cloud London outage last?** Google's revised report gives 6 hours 8 minutes of main impact, from 07:00 to 13:08 London time, with a closure notice at 15:03. Power was restored to the data halls at 08:20; recovery took longer because storage devices had to recover from unclean shutdowns. **What is Google changing after the europe-west2 outage?** The controllers were reconfigured to check undervoltage and breaker status, generator auto-start was tested inside the battery window, emergency procedures gained manual steps practised twice per shift, and Google set out to validate controller configurations at similar sites and froze planned maintenance at other sites with similar topologies. --- Cite the primary source this article names, and link the article as the reconstruction. Corrections are appended above, dated.