Network Troubleshooting as a Stack: Find Which Layer Is Broken First

When an HTTPS request times out, the first step is to determine which layer of the network stack is broken. The network operates as a stack, with each request passing through the same layers in order: Application, TLS, Port, DNS, Gateway, Route, Interface. To troubleshoot, start from the lowest layer and work your way up.

To check the Interface layer, verify that there is a link and an IP address assigned to the network interface. Use the command "ip addr show" to see the interface status. If the interface is not in the state "UP" or lacks an IP address, the issue lies here and no further troubleshooting is needed. If the interface has a valid IP, move on to the next layer.

The Route layer checks if there is a path to the destination. Use the command "ip route get [destination IP]" to see the exact route the kernel would use. This command displays the source IP and the gateway, if any. If the output indicates "Network is unreachable" or there is no default route, the routing layer is the problem.

Next, inspect the Gateway layer to determine if there is a path to the first hop. Use "ping -c3 [gateway IP]" to test reachability. Additionally, use "ip neigh show" to view the ARP table. If the gateway entry is in the "REACHABLE" state with a MAC address, Layer 2 is functioning correctly. If the ping fails or the neighbor entry is not in the expected state, the gateway or its connection to the network is likely the issue.

After addressing the Gateway layer, move to DNS to confirm that the domain name resolves to the correct IP address. Use "dig +short [domain name]" to perform a DNS lookup. A failure to resolve the domain name or receiving an incorrect IP address points to a problem at the DNS layer. Additionally, compare the returned IP address with the expected one, as resolving to a stale address may indicate an application bug.

The next layer to verify is the Port layer. Determine if the service is actually listening on the specified port by using "nc -vz [domain name] [port]". Three possible outcomes indicate different issues: a successful connection, a "Connection refused" message (service not running or listening on the wrong port), or a timeout (firewall or security group blocking the traffic). These distinct responses help pinpoint the exact layer causing the problem.

With the Port layer checked, proceed to the TLS layer to ensure the handshake completes successfully and the certificate is valid. Use "openssl s_client -connect [domain name]:443 -servername [domain name]" to perform a TLS handshake. Look at the "Verify return code" at the end of the command output. A result of 0 means the certificate is valid, while any other code indicates an issue with the certificate chain, expiration, or hostname mismatch.

Finally, the Application layer is the last step. Use "curl -v [domain name]/[endpoint]" to verify that the application responds correctly. Only when all lower layers are functioning properly should application-level errors be investigated. The "-v" flag provides a detailed output, including DNS, connect, TLS handshake, request, and response headers, which can help confirm the layer-by-layer conclusion.

By following this systematic approach, network troubleshooters can quickly identify the broken layer and avoid guessing or making random changes. This method is particularly useful for diagnosing issues like timeouts, as each command targets a single layer with a distinct signature. Remember to always probe and test on systems you have permission to access to maintain ethical and legal standards in network diagnostics.