Cover art for Microsoft traces 365 outage to dormant July bug

Microsoft traces 365 outage to dormant July bug

A defect in July's network gateway update lay hidden for three months before a surge in load triggered a cascade failure across Microsoft's services.

7 cards · 1m 50s · tap to begin

From · · · 1m 50s

Microsoft traces 365 outage to dormant July bug

A defect in July's network gateway update lay hidden for three months before a surge in load triggered a cascade failure across Microsoft's services.

In brief

A defect in July's network gateway update lay hidden for three months before a surge in load triggered a cascade failure across Microsoft's services. Microsoft blamed last week's global outage on a July bug that lay dormant for three months before load and capacity loss triggered a cascade affecting over eighty…

July update hid a latent defect

Microsoft traced the outage to a July update for network gateway servers. The bug lay dormant for more than three months before conditions triggered it.

“A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions.”

Testing missed the flaw in the code

The July update passed load testing and ran worldwide without problems. But the defect caused lock contention inside an internal Windows process, exhausting CPU and memory.

“The load tests focused on weaker hardware with fewer cores. The contention is worse on newer machines with more cores.”

Three conditions collided at once

Three conditions collided at once

Teams failed over to the gateway carrying Exchange Online, raising load. An unrelated issue from the day before had taken nearly fourteen percent of gateway capacity offline.

Cascade failure spread across servers

When one server stopped responding, traffic moved to the next healthy one, which hit the same bug. Within two hours, more than eighty percent of servers were unresponsive.

Recovery was slowed by the defect's age

Microsoft's automated recovery was not designed for such broad failure. Engineers wrote an update to revert the Windows code path to its previous state and applied it to server groups.

“Since the main code defect was a latent bug from several months previous, our normal rollback, flighting, and feature flags were unable to quickly revert the change.”

Services recovered at different speeds

Teams, OneDrive, SharePoint and Office apps moved off faulty gateways and came back sooner. Exchange Online could not for scale reasons, so Outlook stayed down longest.

Microsoft plans full report by November twentieth

The outage hit governments, courts, hospitals and businesses across seventy four countries and territories. Microsoft plans to publish its final post-incident report on twenty November.

What to remember

Microsoft blamed last week's global outage on a July bug that lay dormant for three months before load and capacity loss triggered a cascade affecting over eighty percent of gateway servers.

Read the original on The Next Web

Read the TL;DR — this story on one card.

How we make summaries

React

Sign in to react and comment.

Comments (0)

Life is short. Keep it sweet. Respect others' opinions and be kind!

    Recommended next

    More decks on microsoft and related topics.