The CoreDNS Black Hole: how one dead DNS pod broke our API gateway

Our API gateway experienced intermittent failures, returning an "Upstream Service Unavailable" error to roughly one-third of all requests. The issue persisted despite restarting the gateway, making no change in the backend logs or traffic. The cause was traced to a single open network socket pointing at a non-existent Kubernetes pod.

To understand the problem, imagine a busy office clerk who keeps a line open indefinitely after realizing the intended recipient's extension is no longer active. The same principle applied to the nginx worker process, which held a persistent connection to a dead CoreDNS pod. When Kubernetes replaced the pod, the socket continued pointing at nothing, resulting in the DNS lookup timeout and the subsequent 502 error.

The problem stemmed from three components - nginx worker process, CoreDNS, and Cilium - combining to create a permanent failure point. Addressing the issue required three steps: spreading the traffic evenly across workers, maintaining multiple resolver entries for DNS lookups, and eliminating DNS lookups from the request path altogether.

The first approach involved using the accept_mutex off and reuseport configurations, ensuring the kernel balanced connections evenly across workers. The second step involved listing the same DNS name twice in the resolver directive, creating two independent sockets. Finally, the fix involved moving DNS lookups out of the request path and instead using an upstream group with nginx to pool connections to healthy backend instances.