Reporter’s notebook: Microsoft scrambles as Outlook inbox goes dark

Microsoft found itself in a scramble as its Outlook service went dark, overshadowing other news involving Jensen Huang and additional cheating allegations. One woman asked me if there was a problem with Outlook as we were settling down for the night. Microsoft acknowledged the issue at around 8pm our time and reported that users in the Americas couldn't reach their Exchange Online mailboxes.

Complaints about Outlook peaked at over 138,000, according to Downdetector. Microsoft initially suspected that the overload of requests was the cause and took measures to pause deployments, reroute traffic, and restart servers. However, they later discovered that several servers had unexpectedly dropped out and were struggling under the additional weight.

Another theory was that automated health checks were failing, causing servers to remove themselves from service. A component responsible for processing server health was found to be functioning normally, but it wasn't clear if it was the source of the problem. Microsoft found a temporary solution when a different health-check method brought some servers back online.

However, the restored servers could become unresponsive again once they started receiving traffic, leading to Microsoft implementing traffic limitations and rebuilding a subset of servers on an earlier configuration. The outage affected various Microsoft 365 services, including Copilot, Defender's Safe Links, Purview's data-loss prevention tools, Intune enrolment, Graph API calls, Planner, To Do, and Universal Print.

While some Windows 365 Cloud PCs became unreachable, Teams calls largely remained functional. Unfortunately, calendars weren't accessible, causing concern among users. Microsoft is currently working on identifying the root cause of the issue and restoring backend servers, taking a cautious approach to ensure the implemented changes are effective.

As of now, they are not aware of any malicious activity causing the incident, and they will provide a comprehensive update in a Post-Incident Report.