Kubernetes DNS was failing 33% of the time. CoreDNS was fine.
A WordPress site on our platform started throwing Error establishing a database connection . Not always — maybe one page load in three. Reload, and it worked. Reload again, broken. The obvious suspect is MySQL. It wasn't MySQL. It took us longer than it should have to stop looking there, so here is the whole path, including the two wrong turns. The first wrong turn: blaming the database WordPress…
A WordPress site on the platform intermittently experienced errors establishing a database connection, with success rates around one in three attempts. The problem was not MySQL, as the database connection was healthy. The issue was traced down to DNS resolution, specifically the CoreDNS service. CoreDNS pods were functioning correctly, but the root cause was an issue with the network link between specific nodes.
The problem was due to a stale kernel datapath flow, which resulted in 100% packet loss for pod-to-pod traffic across the affected link. Deleting the affected weave-net pod and allowing the DaemonSet to recreate it resolved the issue, restoring DNS resolution and eliminating the WordPress errors. The incident highlights the importance of investigating all potential causes, including DNS and network issues, when dealing with intermittent failures.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
