Discord's Wild 2026: A 3-Hour Voice Catastrophe, a Hidden Mailbox Monster, and the Biggest Privacy Win in Gaming
A deep dive into the March 25th voice outage and the E2EE rollout that followed — two stories that together define Discord's most technically eventful year yet. If you were trying to hop on a Discord voice call on the afternoon of March 25, 2026, you probably stared at that maddening "Awaiting Endpoint" spinner for a long time. Maybe you refreshed. Maybe you rage-quit. Maybe you blamed your ISP.…
On March 25, 2026, Discord experienced a major voice outage that lasted for over three hours. The incident began with a routine Kubernetes configuration change, which led to a cascading failure in the distributed systems infrastructure. The change involved adjusting pod resources for the session management service, responsible for routing messages, presence updates, and notifications to connected clients.
When the Kubernetes configuration was applied, it terminated 50% of pods in one availability zone, causing a safety check to run longer than the termination grace period. This resulted in the destruction of 17% of active Discord sessions globally, which was a significant shock considering the platform's hundreds of millions of users.
As the pods were terminated, all entities monitoring those sessions received a {:DOWN, ...} message in their mailboxes. This created a tidal wave of messages throughout the system, which primarily affected the gateway service. When sessions died, gateways instructed clients to reconnect, causing millions of clients to simultaneously reconnect to the affected availability zone.
Memory on gateway nodes spiked, leading to OOM kills. As the affected users began reconnecting, the gateway connections were killed by the cascading OOM events, causing them to fail over to other zones.
The real issue stemmed from Discord's voice syncer service, which is responsible for routing voice and video calls. The voice syncers constantly evaluated active voice states across guilds and direct message calls, and sent RPC commands to selective forwarding units (SFUs) to handle disconnections and recreate calls. When sessions disconnected from voice calls, the voice syncers sent RPCs to the SFUs, and when sessions reconnected, they sent RPCs to recreate calls.
With 17% of sessions dropping and reconnecting, the voice syncers had to fire an enormous flood of outgoing HTTPS connections to the SFUs. The voice syncers used an internal library called Holster, which managed connections through an Erlang HTTP client. The problem arose when the Holster.Pool supervisor, responsible for managing the Erlang gun connections, grew to contain a million-message mailbox.
This created a bottleneck, as the selective receive operation in Erlang becomes slower as the message mailbox grows. Consequently, the supervisor mailboxes grew, leading to the bottleneck that compounded under load. As the mailboxes grew, new connection requests began timing out. Moreover, the etcd connection, which powers Discord's Elixir service discovery, also went through the same bottlenecking supervisors, causing further issues.
The outage lasted several hours, highlighting the complexity and fragility of Discord's distributed systems architecture.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
