Skip to content
Trending
BreakingTech Business2 min read

Microsoft 365 outage was caused by a dormant July update defect

A July update sat idle for 109 days. Then two coinciding failures exposed it — and took down Outlook, Teams, and more for over a day across 74 countries.

By The Ledger

The Tech Business Desk · (1 hour ago)

Reported fromoutlet, credited1
Photo via The Next Web

Microsoft has confirmed its 12 November global Microsoft 365 outage — which took down Outlook, Teams, OneDrive, and SharePoint for up to 36 hours in some regions, was triggered by a dormant bug in a July update to its network gateway servers. The defect lay undetected for over three months before cascading failure began at 01:07 UTC on 12 November, according to Microsoft’s preliminary post-incident report, published in the Microsoft 365 admin centre on 15 November and summarized by Charles Lamanna, EVP for Copilot, agents and platform, on X (formerly Twitter) on 16 November.

The company says it will release its final report on 20 November. A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions,” Lamanna wrote. That defect, lock contention inside an internal Windows process, exhausted CPU and memory on newer gateway servers with higher core counts. It passed all load testing because those tests ran on older, lower-core hardware.

Two compounding events then aligned: Teams failed over to the same gateway infrastructure that handles Exchange Online, spiking demand. Simultaneously, an unrelated issue from the previous day had already removed nearly 14% of that gateway’s server capacity. Traffic flooded the remaining healthy nodes, each of which then hit the same bug.

Within two hours, more than 80% of the gateway servers were unresponsive. Recovery stalled because Microsoft’s automated systems weren’t built for this scale of simultaneous failure. And rollback wasn’t possible: “Since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change,” Lamanna said.

Engineers instead wrote and deployed a targeted patch to restore the prior Windows code path, applied in waves starting at 07:20 UTC on 13 November. Services recovered unevenly. Teams, OneDrive, SharePoint, and Office apps moved off the faulty gateways faster. Exchange Online, and So Outlook, stayed down longest due to scale constraints.

Microsoft sent its final customer update at 04:15 UTC on 14 November. More than 1,000 engineers and staff joined incident response calls led by Lamanna and cloud chief Scott Guthrie, Bloomberg reported. The outage affected governments, courts, hospitals, and enterprises across 74 countries and territories, per Downdetector data cited by the Associated Press.

House Homeland Security Committee chair Andrew Garbarino told Politico he is seeking further information from Microsoft and CISA, saying, “We need to understand what happened and how to make these systems more resilient.” Microsoft’s official statement on 13 November ruled out cyberattack and AI agents: “This was a technical issue not related to cyber or AI / agents. The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change.” The company added that restoration was gradual to ensure stability, “While this took longer, it helped ensure a stable restoration.” Last month’s Google Cloud London-region outage, which disrupted Revolut and others, now sits alongside this event as a marker of growing systemic risk in hyperscale infrastructure.

Microsoft’s final post-incident report is due 20 November.

Reported from

How this story was made

Written by the Hitechreports desk from the reporting credited above, with facts attributed to their original publishers. We do not test devices ourselves; anything about performance, battery life or cameras comes from the outlets that did. Prices are as reported at the time of writing. Editorial policy · Report an error

What did you make of it?

Filed by The Tech Business Desk

Big tech, startups, earnings, regulation and policy — the decisions that shape the products.

More from this desk →

Be the first to comment

Join the argument. No password, just your email or a passkey.