Microsoft Outage Traced to Overloaded Servers and Network Change; Root Cause Still Under Investigation

Servers brought back online sometimes went unresponsive again under normal traffic
Microsoft's early fixes restored some servers, but they failed once regular traffic returned.
Mark

Start with the basics. What actually broke?

Mimi

Exchange Online, the cloud system behind mailbox access for Outlook. Microsoft says many servers became unavailable, and the remaining ones took on their workload and strained.

Mark

So it wasn't a flood of users?

Mimi

Right. Microsoft first spoke of unusually high load, then clarified it wasn't a sudden jump in overall traffic. The load was a symptom of lost capacity.

Luke

Worth noting that's Microsoft's own account, and it's still changing. The company says the root cause is under investigation.

Mark

What about the network change being reversed?

Mimi

It coincided with the start of the outage, so Microsoft is trying to roll it back and has paused other deployments and configuration changes.

Luke

Coincided isn't caused. Microsoft hasn't said that change is responsible, and that distinction matters.

Mark

Why did the first fixes fail?

Mimi

Restarts, health-check changes and rerouting brought some regions back, but restored servers sometimes went unresponsive again once normal traffic returned.

Luke

And we don't know what triggers that. Engineers are still analysing it, which is why traffic is being limited in some locations.

Mark

Is it an attack?

Mimi

Microsoft hasn't reported evidence of malicious activity.

Luke

That's absence of a report, not a ruling-out. The 130,000 figure is also Downdetector user reports, not a count of affected customers.

  • More than 130,000 people in the United States reported Outlook trouble at the peak, and reports in India climbed into the tens of thousands as the outage carried into Friday morning.
  • Servers went offline unexpectedly, shifting workload onto the machines still running and pushing them toward overload until they stopped responding.
  • Microsoft paused deployments and configuration changes and is working to reverse a recent network change that coincided with the outage's start, though it has not confirmed that change as the cause.
  • Early fixes restored some servers in certain regions, but they failed again under normal traffic, so Microsoft is limiting traffic in key locations and rebuilding servers on an earlier configuration.
  • The outcome now hinges on whether the rebuilt servers hold steady under real demand, since every previous recovery has faltered at exactly that point.

On a Thursday night that rolled into Friday morning, millions of people who rely on email as a quiet part of daily life found their Outlook inboxes stalling, a reminder of how much of modern work now rests on machines most of us will never see. Microsoft, whose cloud carries mailboxes for vast numbers of organisations across continents, has traced the trouble to a loss of servers that pushed heavier burdens onto those that remained, but it has not yet named the single cause that set this in motion. The company is retracing its steps, pausing changes and rebuilding machines with care, while users wait with the familiar patience of people who know that the systems they depend on are, in the end, made by fallible hands.

More than 130,000 people in the United States reported trouble with Outlook to Downdetector at the height of a Microsoft outage that began on November 12 and was still unresolved into November 13. In India, reports climbed into the tens of thousands as the disruption carried into Friday morning, and users in Australia, Japan and other markets were caught up too.

The trouble centres on Exchange Online, the cloud infrastructure that handles mailbox access for Outlook and supports several Microsoft 365 services. Microsoft first suggested the systems were under unusually high load, then clarified that overall traffic had not surged. Instead, a large number of servers had unexpectedly gone offline, pushing their work onto the machines still running, which were then overwhelmed by a core process and stopped responding.

Microsoft has paused deployments and configuration changes and is trying to reverse a recent network change that coincided with the outage. The company has not said that change is responsible; it remains one line of inquiry among several.

Initial fixes offered only partial relief. Some servers were restored by altering how their health was checked, and others were restarted or had traffic redirected, bringing recovery in some regions. But restored servers often failed again once normal traffic returned. Microsoft is now limiting certain traffic in affected locations and rebuilding some servers on a previous software configuration, with plans to expand that if they remain stable. Its latest update admitted that the serviceability changes were "not taking as expected."

Microsoft has reported no evidence of malicious activity. For now, the question users are watching is whether the rebuilt servers can hold up under everyday traffic, because that is precisely where earlier recoveries broke down.

More than 130,000 people in the United States reported trouble with Outlook to Downdetector at the height of a Microsoft outage that began on November 12 and was still unresolved into November 13. In India, reports climbed into the tens of thousands as the disruption carried over into Friday morning. Users in Australia, Japan and other markets were affected too.

The trouble is centred on Exchange Online, the cloud infrastructure that handles mailbox access for Outlook and supports functions across several Microsoft 365 services. Outlook, Exchange Online and a number of those other services remained disrupted as Microsoft released more detail about what it was seeing inside its systems. The company says it is still investigating the exact root cause.

Microsoft's early explanation was that the affected infrastructure was under unusually high load. It later clarified that this was not the result of a sudden surge in overall traffic. Instead, a large number of servers unexpectedly became unavailable. With those machines gone, more traffic and workload shifted onto the servers still running, which put additional strain on them.

Microsoft then identified another clue: a core process on the affected servers was becoming overwhelmed, causing the servers to stop responding. The lost capacity fed into further connectivity problems for users.

The company also began looking at a small number of recent configuration changes after finding that some may have contributed to the incident. It paused deployments and configuration changes while it examined the possible link. The latest reports indicate Microsoft is trying to reverse a recent network change that coincided with the start of the outage, and is testing configuration changes on affected servers. Microsoft has not said the network change is definitively responsible; it remains part of the investigation.

The first fixes brought only partial relief. Microsoft found it could restore some servers by changing how their health was checked, and it restarted servers and redirected traffic to alternative infrastructure. That produced some recovery in certain regions. But servers that had been brought back sometimes became unresponsive again once they began receiving normal traffic.

Because of that, Microsoft is taking a more cautious approach. It is temporarily limiting certain types of traffic reaching affected infrastructure in specific locations while engineers analyse what is causing the servers to fail. It is also rebuilding some affected servers using a previous software configuration, and plans to extend that approach to more infrastructure if the machines stay stable under traffic.

In its latest update, Microsoft said the changes meant to improve serviceability were "not taking as expected." Engineers are now working on other ways to add capacity while dealing with continuing imbalances in how traffic is managed.

On the question many users will ask, the company's investigation has focused on server availability, traffic management, configuration changes and the overloaded server process. Microsoft has not reported evidence of malicious activity behind the disruption. The thing to watch is whether the rebuilt servers hold up under normal traffic, since that is exactly where the earlier recoveries fell apart.

The changes meant to improve serviceability were "not taking as expected."
— Microsoft, latest update
Quieres la nota completa? Lee el original en News18 ↗
Contáctanos FAQ