Microsoft’s Own AI Service Took Down Azure for Nearly 8 Hours — Because of a Change That Looked Fine in One Region

By Oleg

At 09:20 UTC on May 29, 2026, an internal change to how Microsoft’s cloud infrastructure surfaces capacity-related failures started causing trouble in a single Azure region, Australia East. By the time Microsoft fully contained it at 17:05 UTC — nearly eight hours later — the same rollout had been pushed to a second, much busier region while engineers were still investigating the first, turning a regional blip into a multi-continent outage of the Azure OpenAI Service.

The postmortem timeline Microsoft published is unusually specific, and it’s worth walking through step by step, because the interesting failure here isn’t the initial bug. It’s the decision, made mid-incident, to keep rolling the same change forward before anyone had confirmed what it was actually doing.

The Timeline

The trigger was an upstream change that altered how certain capacity-related failures were surfaced internally — not a capacity shortage itself, but a change in how the system reported and reacted to one. That altered reporting caused a rapid, unexpected increase in internal retry traffic: services detecting what looked like transient failures and retrying more aggressively than the system was built to absorb. That retry storm overwhelmed a shared inference load balancer, the component responsible for distributing AI inference requests across backend capacity.

Automated monitoring caught the drop in request success rates at 09:39 UTC, nineteen minutes after impact began, and incident response kicked off. By 12:17 UTC, engineers had used early crash diagnostics to identify a component they suspected was contributing to the instability and disabled it — a plausible fix, and one that presumably looked like it was addressing the problem.

Then, at 14:20 UTC, the same upstream rollout that had caused the original trouble in Australia East reached Sweden Central. Sweden Central carries significantly higher traffic volume than the initial affected region, and the same retry-amplification dynamic hit a shared load-balancing layer under much heavier real load. The disabled component from two hours earlier hadn’t fixed the underlying cause — it had only addressed a symptom in the original region — and the second rollout re-triggered the same failure mode at a scale the first incident hadn’t prepared anyone for.

It took until 16:30 UTC — over two hours after the Sweden Central escalation, and seven hours after the original detection — for engineers to actually identify the source of the amplified retry traffic, isolate the offending workload onto dedicated infrastructure, and begin coordinating a rollback of the triggering change with the upstream team. Impact was confirmed mitigated by 17:05 UTC.

Why the Sequence Is the Real Story

Every large-scale outage postmortem has a root cause, and “retry amplification overwhelming a shared load balancer” is a familiar one — it’s a well-understood failure mode in distributed systems, not a novel discovery. What makes this incident worth examining isn’t the mechanism, it’s the sequencing: the same change that was already under active investigation as a suspected cause of instability continued rolling out to additional regions while that investigation was ongoing. Whatever gating existed between “this rollout is suspected of causing an incident in Region A” and “this rollout should be paused everywhere until we understand it” either didn’t fire, or the connection between the Australia East symptoms and the in-flight Sweden Central rollout wasn’t made until after the second region was already affected.

That’s a coordination and change-management failure sitting on top of the technical one. The retry-storm mechanism explains why the outage happened at all. The decision to keep deploying the same change during an active, unresolved incident explains why it got dramatically worse partway through, rather than resolving after the first region’s mitigation.

What Makes This an AI-Infrastructure Story Specifically

The system that got overwhelmed wasn’t generic Azure compute — it was the Azure OpenAI Service’s inference routing layer, the shared infrastructure that customer applications hit every time they send a request to a hosted AI model. That’s a meaningfully different blast radius than a single product outage: any application built on Azure OpenAI Service, across every customer using it in the affected regions, inherited the latency, timeouts, and 5XX errors simultaneously, for the duration of the incident. Reports of impact concentrated in Europe and Australia East track directly with which regions actually received the problematic rollout.

Microsoft has since said it’s implementing stronger overload prevention and workload throttling controls, with an estimated completion date in July 2026, alongside ongoing work to remove single points of failure in the inference routing layer specifically. Both responses target the technical failure mode. Neither directly addresses the sequencing failure — continuing a risky rollout mid-investigation — which is arguably the more procedurally interesting lesson buried in an otherwise ordinary infrastructure postmortem.