On a Thursday night in November, the engineers and executives of Microsoft gathered in dark rooms and basements, hunting for a way to bring back the email and collaboration systems that much of the working world leans on. A flawed network upgrade had quietly taken servers offline, and the failure rippled outward from the Americas to hospitals, schools, businesses and congressional offices across the globe. It is a reminder, felt once again, that the so-called cloud is only a set of finite machines, and that the companies we trust with our daily work can wound themselves as surely as any outsid
Inside Microsoft's Overnight Scramble to Restore Outlook and Services After a Global Outage
Hospitals, schools, businesses and Congressional offices lost email and other services for hours; the article reports no casualties.
The company's own systems were to blame.
Mark
Why does this one stand out? Outages happen.
Mimi
Because of what was down. Email is still how work gets done, and hospitals, schools and Congressional offices were affected. Microsoft's own staff couldn't use it.
Mark
What was the cause?
Mimi
Microsoft says a recent network upgrade shut down some servers. It first suspected a traffic spike, and it ruled out a cyberattack.
Luke
Worth noting that's Microsoft's own account. The reporting doesn't describe an independent review of the cause, and it doesn't say what exactly in the upgrade failed.
Mark
What was the overnight call about?
Mimi
Scott Guthrie proposed fixes and put Outlook first. The group debated shutting off other services so a "stampeding herd" of demand wouldn't knock systems down again as they recovered.
Luke
That scene comes from one unnamed person who wasn't authorized to speak publicly. It's a single account of the call.
Mark
And the business angle?
Mimi
Microsoft 365 has more than 450 million users and brought in almost $28 billion last quarter. William Blair analysts call the financial hit manageable but see reputational risk.
Luke
That's a forecast, not a result. Whether any customer actually reassesses is unknown, and the article reports no casualties or measured costs from the outage.
Le Pouls
- Outlook, Teams and about twenty other services went dark, leaving hospitals, schools and Congressional offices without email for hours.
- Servers that came back online kept shutting down again, forcing engineers into a frantic cycle of restarts and strain on the system.
- Leaders debated which services to switch off so that a surge of returning demand would not crash the recovery a second time.
- Microsoft ruled out a cyberattack, blamed its own network upgrade, and restored services gradually, with stability confirmed at each stage.
- Analysts warn that enterprises may now rethink how heavily they depend on Microsoft's infrastructure, a threat to its cloud and AI ambitions.
On a Thursday night in November, the engineers and executives of Microsoft gathered in dark rooms and basements, hunting for a way to bring back the email and collaboration systems that much of the working world leans on. A flawed network upgrade had quietly taken servers offline, and the failure rippled outward from the Americas to hospitals, schools, businesses and congressional offices across the globe. It is a reminder, felt once again, that the so-called cloud is only a set of finite machines, and that the companies we trust with our daily work can wound themselves as surely as any outside attacker. What customers choose to do in the months ahead will reveal how deeply that lesson has taken hold.
Late on a Thursday night, hundreds of Microsoft engineers and senior executives joined one video call from darkened offices and home basements. Outlook, the email service much of the working world depends on, was down, and so were many other things. The trouble had begun that morning on the West Coast and spread quickly across the Americas and beyond. Hospitals, schools, businesses and congressional offices in Washington lost access, and even Microsoft's own employees could not reach their inboxes. About twenty services were ultimately affected, including Teams and the Copilot assistant.
At first the company suspected a sudden surge in traffic. It soon found that many servers in its data centers had gone down, and when some returned, they failed again, like a stubborn game of whack-a-mole. On the overnight call, leaders weighed how to switch services off and on so that returning systems would not be overwhelmed by a stampeding herd of new demand. Participants rubbed their foreheads and chewed their nails as they debated. Executives had already apologized and insisted that no cyberattack was responsible; the fault lay in Microsoft's own recent network upgrade.
By Friday morning, services were steadily improving. Microsoft said it had restored them gradually, checking stability at every step, a slower path but a sturdier one. The episode fits a pattern: last year an Amazon cloud outage lasted fifteen hours, and in 2024 a faulty software update grounded airlines and delayed surgeries. Microsoft 365, used by more than 450 million people, is also central to its push into artificial intelligence, which makes the outage's lesson sharper. Consolidating so much work with one vendor can create a single point of failure. Analysts expect the short-term financial cost to be manageable, but the reputational damage may lead customers to reconsider how much they depend on Microsoft's infrastructure.
Late on Thursday night, hundreds of Microsoft engineers and senior executives sat in darkened offices and home basements, all logged into one video conference. The subject was the kind of scenario a technology company dreads: Outlook, the email service much of the working world depends on, was down, and so were many other things.
The trouble had started that morning, around 9 a.m. on the West Coast, where Microsoft is headquartered. The Outlook problems spread from the Americas around the world. Hospitals, schools, businesses and Congressional offices in Washington were affected. Even Microsoft's own employees could not use email. Microsoft later acknowledged that about 20 different services had been impaired, including Teams and its A.I. assistant Copilot.
The company's first theory was an unexpected spike in traffic. It then determined that a large number of servers in its data centers were down, putting strain on the rest of the system. Some servers were brought back, only to shut down again, a pattern the article likens to a game of Whac-A-Mole.
On the overnight call, according to a person familiar with it who was not authorized to speak publicly, Scott Guthrie, the senior-most executive overseeing Microsoft's computing infrastructure, proposed possible fixes. He said that although other systems were also down, teams should put Outlook first. Charles Lamanna, who heads many of the company's productivity services, was on the call too. He had already apologized publicly and said a cyberattack had been ruled out. The company's own systems were to blame. Participants chewed fingernails and rubbed their foreheads as they debated how to switch off other services, so that systems coming back online would not be hit by a "stampeding herd" of new demand and collapse again.
By Friday morning, Microsoft had apologized at least twice. It said the cause was a recent upgrade to its network that shut down some servers, and that systems were steadily improving. "Given the scale of demand, we restored services gradually, validating stability at each stage," the company said, adding that this took longer but ensured a stable restoration.
The episode is the latest reminder that modern life and business run through a handful of large companies, and that those companies can wound themselves with flawed updates as readily as outsiders can attack them. Last year, Amazon's cloud services were down for 15 hours, knocking out apps and sites including Zoom and parts of The New York Times. In the summer of 2024, a faulty software update from the cybersecurity firm CrowdStrike grounded airlines, closed businesses early and forced surgeries to be rescheduled. Luca Arnaboldi, an assistant professor of cybersecurity at the University of Birmingham in England, noted that "the cloud" really means servers with finite resources.
The affected products belong to Microsoft 365, a cloud subscription business used by more than 450 million people that brought in almost $28 billion in the quarter ended Sept. 30. Online memes suggested renaming it Microsoft 364. The business also underpins Microsoft's A.I. push: the company argues that customers who keep more data on its systems can adopt A.I. tools more easily. The outage showed the flip side, that consolidating vendors can create a larger single point of failure.
Analysts at the investment bank William Blair wrote Friday that the short-term financial fallout should be manageable. The reputational damage is the larger concern, they said: "We believe the outage could prompt enterprises to reassess their dependence on Microsoft's infrastructure." What customers decide in the coming months will show whether that warning holds.
Citations marquantes
We believe the outage could prompt enterprises to reassess their dependence on Microsoft's infrastructure.— William Blair analysts, research note
Given the scale of demand, we restored services gradually, validating stability at each stage.— Microsoft statement
Envie de l'histoire complète ?
Lire l'original sur The New York Times ↗