Connection refused" has three different root causes here's how to tell them apart with evidence

When a connection attempt fails and returns a "Connection refused" error, it is easy to confuse the issue with a silent timeout. However, these two problems have different root causes and affect the system at different layers of the network stack. Ignoring this distinction can lead to unnecessary restarts and fixes that do not address the actual issue.

To help diagnose the problem, it is useful to understand the naming convention for network failures: Name (DNS) → Route to IP → Port open on target → Filter in the path → App responds. By breaking down the problem layer by layer, you can determine the exact cause.

The first layer to examine is the DNS resolution. If the DNS name cannot be resolved, the client will not be able to reach the server even if the network and port are otherwise functional. To test this, you can modify the `/etc/resolv.conf` file inside a container to force a DNS failure. If the DNS resolution issue is the culprit, the client will be able to access the server by using its IP address directly.

The next layer to check is whether the port is open on the target server. If the port is closed, a "Connection refused" error will be returned immediately. To verify this, you can use the `ss` command to check if any process is listening on the specified port. If there is no output, the port is indeed closed.

The third layer is the firewall rules. If the firewall is dropping incoming traffic on the target port, the error will be a silent timeout, with no error message returned to the client. To test this, you can add an `iptables` rule to drop incoming traffic on the specified port. If a silent timeout occurs, it confirms that the firewall is the issue.

The final layer to consider is the application itself. If the application is not responding to requests, you will see a "Connection refused" error. However, this error will be accompanied by a TCP RST packet, which can be seen using `tcpdump`. If you see a RST packet, it confirms that the application is not responding.

These same three failure modes are present in Kubernetes, but they are labeled differently. A misconfigured CoreDNS can cause DNS failures, a Service with empty Endpoints can lead to port closures, and a NetworkPolicy silently dropping traffic can result in silent timeouts.

In practice, it is often most efficient to reach for tools like `tcpdump`, `dig`, and `nc` early in the debugging process. These tools allow you to examine the network traffic at each layer, making it easier to isolate the issue. However, it is important to note that sometimes the symptom can actively lie, such as a REJECT rule appearing to be a closed port. By carefully examining the evidence, you can determine the true cause of the issue and take appropriate action to resolve it.