Load Balancing Starts Before the Load Balancer: Routing Patterns for Distributed Systems

This article is the third in a series on control plane architecture and design patterns. In the previous articles, the focus was on a high-level overview of control planes and scaling components. This piece delves into hot partitions, which occur when a single piece of a system experiences disproportionate traffic, despite well-provisioned infrastructure. This issue lies at the heart of load balancing and routing, which are more important than load balancers themselves, especially in distributed systems.

For a simple web application, load balancing can be achieved using DNS-based load balancing, where multiple A records point to different IPs, and clients or resolvers distribute requests across them. However, this approach is coarse-grained, providing little control over traffic distribution, and has limited real-time awareness of backend health. Health checks with DNS providers can help identify unhealthy IPs, but they come at a cost and have no global view of the system.

A dedicated load balancer or proxy, such as an ALB, NLB, nginx, or Envoy, can be placed in front of the fleet, actively health-checking backends and routing each connection or request according to an algorithm. For stateless, interchangeable backends, this approach covers most of the load balancing needs. However, this model falls short when backends are not stateless and interchangeable, as the routing decision must be coupled to where the data lives, which is a placement problem rather than a traffic-distribution problem.

Placement strategy becomes load balancing in advance when creating a new table or dataset in a system like Cassandra. The control plane decides how partitions get distributed and replicated across the fleet. An improper placement strategy can create imbalances even before any request arrives. Good placement strategies aim to spread partitions evenly across available hosts, respect replication and fault-domain constraints, and account for heterogeneous hardware.

Despite a well-designed placement strategy, hot partitions can still occur. This happens when traffic to specific keys is wildly uneven due to a poorly chosen partition key. Detecting and correcting for hot partitions requires per-partition metrics, threshold models to distinguish overloaded partitions from normal variance, and attribution to know which keys within a partition drive the load.

Corrective actions, such as re-partitioning live, are delicate operations that the control plane must perform carefully to avoid outages.