ECMP Hash Polarization: Why Two Identical Links Carry 88% and 12%

Equal-cost multi-path routing (ECMP) is a crucial technique in modern data centers, allowing multiple paths between leaf and spine switches to carry traffic equally. When multiple equal-cost routes exist in a router's forwarding table, ECMP spreads the traffic across all the links, optimizing the use of available resources. However, this elegant solution can sometimes lead to unexpected consequences due to a phenomenon known as hash polarization.

Hash polarization occurs when the deterministic hash function used in ECMP selection is identical across all switches in the fabric. This results in correlated flows consistently taking the same relative path at each tier of the network, causing traffic to polarize onto a few links instead of evenly distributing across the entire fabric. This behavior can lead to hot spots, link congestion, and reduced overall utilization.

The hash function used in ECMP typically takes the 5-tuple of the flow—source IP, destination IP, protocol, source port, and destination port—as input and produces a hash value that determines the egress link. The key properties of a good hash function are determinism and distribution. Determinism ensures that the same flow is always routed to the same member, preventing packet reordering. Distribution aims to spread the traffic evenly across all members, but in the case of hash polarization, this balance is disrupted.

To mitigate hash polarization, several strategies can be employed. One of the most effective solutions is to introduce variation into the hash function by using different seed values or offset values on each switch. This ensures that the same flow will be assigned to different members on different switches, breaking the polarization and promoting a more even distribution of traffic.

Additionally, varying the hash inputs by tier can also help maintain a balanced load. This can be achieved by including different fields, such as the ingress port, in the hash calculation at different layers of the network.

In overlay networks such as Virtual Extensible LANs (VXLAN), ECMP may face challenges due to the encapsulation of the inner flows. If the outer headers, such as the source port, remain constant across all inner flows, the underlay ECMP hash may incorrectly assume a single giant flow and pin it to a single link. To address this issue, it is recommended to introduce UDP source-port entropy, as specified in RFC 7348, by hashing the inner 5-tuple to determine the outer source port.

This ensures that each inner flow is treated as a distinct entity and distributed appropriately across the available paths.

When troubleshooting lopsided links, it is essential to focus on the distribution of flows rather than just the bytes of traffic. A few elephant flows can disproportionately affect a member link, while smaller flows may appear to be insignificant. By counting the number of flows per member link, rather than just monitoring the byte utilization, network administrators can identify the true source of the imbalance.

Additionally, checking the maximum number of paths supported by the platform and ensuring that the installed members align with the number of existing equal-cost paths can also help identify and address ECMP-related issues.

In conclusion, ECMP is a powerful tool for optimizing network performance and resource utilization in modern data centers. However, understanding and mitigating the effects of hash polarization is crucial to ensure the fabric operates efficiently and reliably. By introducing variation into the hash function, varying the hash inputs by tier, and addressing issues specific to overlay networks such as VXLAN, network engineers can harness the full potential of ECMP and avoid the pitfalls of hash polarization.