From The Verge · Tom Warren · · 3m
The bug that broke Outlook for 24 hours
A latent defect in Microsoft's networking stack, combined with a Teams failover and reduced capacity, took down Exchange Online for millions worldwide.
In brief
A latent defect in Microsoft's networking stack, combined with a Teams failover and reduced capacity, took down Exchange Online for millions worldwide. A dormant bug in Microsoft 365 networking gateway servers, triggered by a Teams failover and reduced capacity, caused a twenty four hour Outlook outage affecting…
Outlook goes down across the world

At nine AM last Thursday, Microsoft employees arrived for a normal day.Within an hour, Exchange Online servers powering Outlook began failing in Switzerland, spreading globally.Nearly eighty percent of Exchange servers became unresponsive within hours.
Microsoft marks it catastrophic
Microsoft's internal incident tracking system marked the outage as sev zero, the highest level indicating catastrophic service disruption. Hundreds of engineers began investigating what went wrong and how to restore service.
Failover systems spread the problem
Microsoft has systems designed to route traffic toward healthy servers during failures. In this case, the failover systems worked but also spread the problem, shifting email traffic to overwhelmed servers elsewhere.
Servers stuck in a reboot loop
Some servers entered maintenance mode due to unusually high CPU loads.When engineers tried to bring them back online, they were immediately overwhelmed by waiting traffic.
Not a cyberattack, but a bug
As the outage continued, Microsoft investigated whether it was a cyberattack or related to AI coding.Crash reports and logs quickly ruled out an attack.Engineers discovered an unusual bug in the networking stack of the servers.
A bug dormant since July
The bug had been sitting on Microsoft 365 networking gateway servers since July without any impact.It only became obvious during this major outage because of the specific traffic mix and hardware conditions that triggered it.
Lock contention in Windows process
Charles Lamanna, leader of Microsoft's Copilot, Agents, and Platform team, described the bug as a latent defect causing servers to become overwhelmed under certain load conditions.
“A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions.”
Three problems hit at once
Fifteen hours before the Outlook outage, Microsoft Teams failed over from an alternative networking gateway to the main gateway supporting Exchange Online.
Recovery complicated by cascading failures
It took Microsoft nearly twenty four hours to stabilize Exchange servers.Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again.
“Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again.”
A thousand engineers on one call
Microsoft engineers maintained a running bridge call throughout the outage.At one point it reached the maximum of one thousand participants, showing how many teams were involved in finding the root cause of this huge outage.
Microsoft must improve testing
Microsoft admits the bug went undetected because its load testing focused on weaker hardware with fewer cores, while the bug worsens on machines with more cores and newer hardware.
The headline
A dormant bug in Microsoft 365 networking gateway servers, triggered by a Teams failover and reduced capacity, caused a twenty four hour Outlook outage affecting millions globally.
Read the TL;DR — this story on one card.





