A July patch took down Microsoft 365 email for a day
Microsoft blames monster email outage on software defect introduced in July fix

A software defect introduced in a July 2026 update to Microsoft's network gateway servers lay dormant until a Teams failover and a 14 percent capacity loss collided in November. More than 80 percent of gateway servers went unresponsive, and with the defect months old, engineers couldn't simply roll back — they had to write a patch for the patch. Exchange Online stayed down far longer than services like Teams and SharePoint, which could fail out of the faulty gateway.
Our load testing tended to focus on weaker hardware with fewer cores. The contention issue is worsened on machines with more cores and newer hardware.
- mike_hearn
No big surprises here. Big outages at big firms seem to always look like this.
The electricity industry has this notion they call a black start, where they bring the grid online from zero. It's a very difficult operation because you can't start a generator straight into an offline grid otherwise it gets hit with full demand and can't keep frequency in the right range, which can damage equipment. So they have to start by disconnecting all the loads and then bringing load blocks back in sequence, trying to roughly balance available generation with the size of the next block. In practice black starts are extremely rare, but they can happen after a cascading failure in which something causes some generating stations to trip, redirecting load onto the remaining stations which can't handle it causing them to trip in turn. The entire grid can collapse in just 60 seconds or so once that kind of runaway failure starts.
This kind of outage is very analogous to a black start. A few servers get into trouble - maybe they're in a crash loop or maybe their caches are cold - the load balancers shift traffic onto the remainder which then can't handle the load either and so the failure cascades. Once a service is in this state sometimes the only remedy is to disconnect all load, bring the servers back to stability, then bring load blocks back online in small chunks, making sure to never hit anything with the full load immediately to ensure any caches or JIT compilers have time to warm up.
It's […]
- sinuhe69
No one has asked the obvious question: how much AI contributed to the epic failure by Microsoft. Was the patch programmed and tested by AI? Not because they are inherently bad but it begs the question of how much do the remaining humans at MS understand the patch? And did using AI contribute to the atrophy of knowledge and skills, especially deep, specific skills such as network failure inspection, load balancing and fail over recovery, that led to a slower response and recovery time?
Without a functioning intuition, discovering the causes to initiate recovery effort in a high-stress, unfamiliar situation with catastrophic dimensions could be significantly hampered.
They are valid questions and of particular interest in the age of AI. Productive, highly specialized skills could degrade faster than one imagine once we start rely too much on external helps.
- andix
There is one thing I don't get with this issue. The problem was basically that their system couldn't handle the normal load. Why couldn't they just reduce the load at some point just by dropping/limiting connections?
For example just accept connections from specific (random) IP ranges. They could even cycle through different ranges every 5 minutes to give clients a chance to sync at least once every hour.
The reduced load could be used to gradually bring up servers again without failing, add additional capacity, and gradually restoring access. Completely without fixing the bug at all. The system managed to handle usual load even with the bug, it just hit some critical point that made it collapse. Adding more resources would also fix that.
- malkia
Lock contention, especially cache line contention in large NUMA systems. If hardware was upgraded to NUMA machines, or VM's were not cordoned each to sit in each it's own NUMA node (CPU, Caches & Memory) then something like this would happen.
We saw with our own tools back when an artist were given a low-core "server" like machine with many CPUs (good we thought), and 2 NUMA nodes, and suddely the level editor for our game was running much slower. We could blame std::shared_ptr<> all day long, or similar but it's there and it was real.
Tried the 4kb interleaved paging, but then everything was equally slow. At the end we simply blocked the App to run only on one NUMA node (still using memory from the other one), and things were back to "normal". "Normal", because promises were not kept - we could not use all CPUs.
- mindwok
The biggest surprise here is that they're using Windows for critical network infrastructure. I'm surprised a Windows process has any role here outside the Exchange servers themselves.
- nashashmi
Original source information: https://x.com/clamanna/status/2122209697226948729
Better summary: https://www.neowin.net/news/microsoft-finally-explains-why-o...
- ncr100
They talk about their relatively low-end testing environment PASSing the July code, and being different from their high-end production environment machines, sadly hiding the concurrency bug during the test phase.
Is this commonly acceptable? I'd imagine it's a costly trade-off?
- colonCapitalDee
Seems like a pretty standard big company severe outage to me. These things happen!
- hmokiguess
> The engineers who worked feverishly to restore services deserve commendation and will hopefully never have to buy their own beverages in Redmond again (or at least escape the next round of firings).
What does this mean? I didn't understand
- k0morebi
Not a network engineer so genuinely curious how these things are handled at the epic scale of 365, azure, aws and so on.
Extent of my knowledge is nginx or traefik load balancing which at this scale is virtually equal to no knowledge. But seems like there would be systems in place to simply drop a sliding percentage of excessive traffic or block some regions once critical thresholds are reached rather than compounding the problem with forwarding the growing number of requests downstream to next victim server.