From Ars Technica · · 2m
Three failures combined to crash Microsoft 365
A hidden bug dormant since July, traffic shifts, and reduced capacity combined to take down Microsoft's cloud services for nearly 24 hours.
In brief
A hidden bug dormant since July, traffic shifts, and reduced capacity combined to take down Microsoft's cloud services for nearly 24 hours. A dormant defect in Microsoft's cloud gateway, exposed by traffic changes and reduced server capacity, caused a global outage affecting Microsoft 365 and Exchange for nearly 24…
A hidden bug took down Microsoft 365

A defect in Microsoft's cloud gateway, dormant since July, combined with traffic changes and reduced server capacity to cause a global outage affecting Microsoft 365 and Exchange infrastructure.
The outage lasted nearly 24 hours
Microsoft's preliminary report places the outage between November 12 at 16:20 UTC and November 13 at 16:10 UTC, just under 24 hours total. Additional monitoring remained in place through November 14.
A July bug went undetected in testing
A change deployed to Microsoft's cloud gateway in July created a bug where CPU and memory resources could become exhausted and unresponsive.
Traffic shift exposed the latent bug
About 15 hours before the outage, traffic from Teams was shifted from an alternate gateway to the primary gateway where Exchange Online access is maintained.
Server capacity was already compromised
An existing fault meant Microsoft's gateway server fleet was operating at 14 percent below optimal conditions.
More than 80 percent of gateways froze
As US business operations hit their daily peaks, gateway servers began to freeze. Backup systems that shifted requests away from unresponsive servers encountered the same lockouts on supposedly healthy backup gateways, creating a snowball effect…
Recovery required targeted system updates
Restarting gateway servers was ineffective because large request queues were still waiting for access, immediately triggering the same resource exhaustion.System rollbacks were equally ineffective because the bug from July was still undiscovered.
Microsoft plans full report and safeguards
Microsoft expects to publish its full final incident report on November 20.The company has added additional capacity and traffic-distribution guardrails to avoid similar outages.All cloud services appeared online and operational as of Monday morning.
In short
A dormant defect in Microsoft's cloud gateway, exposed by traffic changes and reduced server capacity, caused a global outage affecting Microsoft 365 and Exchange for nearly 24 hours starting November 12.
Read the TL;DR — this story on one card.


