Diagnosing Packet Loss: Telling Link Errors, Queue Drops, and Rate Limiting Apart
Disclosure up front: I run IPYE, a company that leases IP space. It gets one short paragraph at the end; the rest is plain networking. "We are seeing packet loss" is a report, not a diagnosis. Loss has several distinct causes (corrupted frames on a bad cable, a full queue, a policer, an overloaded CPU, a failing path further away), and each needs a different fix. The most valuable first step is…
Packet loss can have various causes, such as corrupted frames, full queues, policers, overloaded CPUs, or failing paths. To diagnose the issue, one must first determine where the packets are disappearing and the type of disappearance. This article outlines a method for identifying packet loss using common tools on Linux hosts and routers.
Step 1: Measure packet loss properly by running a sufficient number of probes in both directions with multiple packet sizes. Ping and MTR can be used to compare small and large packets. If only large packets are lost, the issue may be related to MTU or fragmentation rather than congestion. Running the same test over IPv6 can help identify path or policy differences.
Step 2: Examine the MTR output carefully. Loss shown at an intermediate hop only matters if it persists to the following hops. Real loss on a forwarding path will appear at that hop and every hop after it, including the destination. Ignore the loss at intermediate hops if it stops at the final hop, as this is likely due to control-plane rate limiting rather than real loss.
Step 3: Inspect your own host first for potential issues. Check the local interface and kernel counters using commands like ip -s link show, ethtool -S, nstat -az, and top or mpstat -P ALL 1. These counters will help identify errors, drops, or overflows that may be causing packet loss.
Step 4: Distinguish between congestion and errors. Errors (CRC) will grow steadily regardless of load and are specific to a single link, while congestion grows with load and usually appears as queue drops on a specific egress interface. Congestion can be checked by monitoring utilisation, queue discipline, and burst sizes. Policing, on the other hand, shows a hard ceiling where throughput plateaus at an exact rate, and loss appears even when the link is not full.
Step 5: Localise the loss along the path by capturing on both ends and comparing packet counts. Use tools like tcpdump to analyze the data and identify which packets were lost or delayed. Test to intermediate points you control to narrow down the segment where the loss is occurring. Reach out to neighbours and upstreams to check interface counters on the specific link, providing timestamps, addresses, and measured rates.
Step 6: Finally, rule out common local culprits such as firewall or conntrack limits, reverse path filtering, duplex or speed mismatches, and CPU saturation. Document the test method, packet size, time, rate of loss, interface counters before and after, and the hop where the pattern begins. This information will help you or a colleague repeat the test after a fix and provide evidence to a carrier if needed.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
