Adaptive Concurrency Limits in Go: Dynamically Protecting Dependencies Under Load

In the realm of software services, a critical aspect often requires attention: the notion of concurrency limits. Initially, developers might opt for a straightforward approach, setting a constant limit such as 'const maxConcurrentRequests = 100'. This creates a semaphore to ensure no more than 100 requests execute at once concurrently.

While effective initially, the scenario changes as the service scales up its CPU resources, the downstream database slows down, new deployments alter request processing times, and traffic patterns fluctuate throughout the day. Eventually, the limit that served well yesterday may become overly restrictive today, or conversely, too lenient might allow dependencies to become overwhelmed. Static limits, while simple to apply, demand constant adjustment.

Imagine a Go service interacting with an external market data API. Initially, a concurrency limit of 50 is set: 'limiter := NewFixedLimiter(50)'. Under normal conditions, the dependency responds in 40 milliseconds, and everything seems healthy. However, as traffic increases, the dependency's processing slows down, with response times escalating to 150ms, then 300ms.

Despite the higher latency, the service continues to permit 50 concurrent requests. The crux of the issue lies in the interdependence between concurrency and latency. As request completion times elongate, existing requests retain slots for longer periods, leading to queue growth and further latency escalation. The challenge is discerning between a dependency with increased capacity, one nearing saturation, one already experiencing queueing, or one recovering from overload. Adapting the concurrency limit automatically based on dependency behavior proves more resilient.

The concept of adaptive concurrency limiting emerges here. Rather than adhering strictly to a predefined number, the system monitors request latency and failure signals, gauges queuing occurrences, and subsequently adjusts the number of concurrent requests permitted. This approach offers a more nuanced solution. This article delves into the following topics: The rationale behind the necessity for dynamic concurrency limits; The interplay between concurrency, throughput, and latency; The inspiration drawn from TCP congestion control for adaptive concurrency algorithms; The implementation of a latency-aware concurrency limiter in Go; The integration of this limiter into an HTTP client; The implications of adaptive limits with backpressure, bulkheads, and retry budgets; The necessary monitoring and testing before deploying adaptive concurrency in a production environment.

Firstly, the problem with fixed concurrency limits is laid bare. Consider a Go service communicating with an external market data API, initially configured with a concurrency limit of 50: 'limiter := NewFixedLimiter(50)'. Under normal circumstances, the dependency responds swiftly in 40ms. As traffic burgeons, the dependency's performance deteriorates, with response times expanding to 150ms, then 300ms.

Despite this, the service still allows 50 concurrent requests. The underlying issue is the relationship between concurrency and latency. When requests take longer to complete, existing requests hold concurrency slots for extended periods, causing new requests to pile up, queues to expand, and latency to worsen further. A fixed limit cannot differentiate between a dependency with enhanced capacity, one approaching saturation, one already within queueing parameters, or one recuperating from an overload.

While manual adjustments to increase the limit are possible, this risks overloading the dependency. Conversely, decreasing the limit might unnecessarily diminish throughput. Finding an ideal concurrency level that maintains high throughput without fostering excessive queueing constitutes the main challenge.

Rate limiting and concurrency limiting serve distinct purposes. A rate limiter regulates the number of requests that can initiate within a specified timeframe, whereas a concurrency limiter governs the number of requests concurrently in progress. Two scenarios illustrate this disparity: In the first scenario, a 'healthy dependency' receives 500 requests per second with an average latency of 40ms, yielding an approximate concurrency of 20.

In the second scenario, a 'slow dependency' also handles 500 requests per second but experiences an average latency of 200ms, resulting in an approximate concurrency of 100. Despite the identical request rate, the second situation demands five times more concurrent requests to sustain the same throughput. This relationship, encapsulated by Little's Law (L = λW), where L represents the average number of items in the system, λ denotes the average arrival rate, and W signifies the average time spent in the system, clarifies why a fixed RPS limit might prove insufficient in safeguarding a dependency.

A service may acknowledge an acceptable number of requests per second while simultaneously amassing too many in-flight operations due to increased response times. Adaptive concurrency, focusing on this relationship, provides a more tailored solution.