ADR: EKS Provisioned Control Plane and 40x HPA Concurrency

On July 28, 2026, AWS revealed that the EKS Provisioned Control Plane enhances Horizontal Pod Autoscaler (HPA) sync concurrency to up to 40 times the default Kubernetes rate, with no changes needed from users. For clusters handling hundreds or thousands of HPA objects, this represents a significant improvement, removing a structural bottleneck that previously made metric-driven autoscaling unreliable under heavy load.

Kubernetes' Horizontal Pod Autoscaler functions through a control loop, evaluating HPA objects every 15 seconds by default, querying metrics, calculating replica counts, and updating Deployments or StatefulSet objects. A crucial detail often overlooked is that this loop's concurrency is limited by the --horizontal-pod-autoscaler-sync_period parameter and evaluation concurrency. In standard Kubernetes, this concurrency is intentionally low due to concerns about saturation on shared control planes.

Financial environments often run between 200 and 800 active HPA objects in a single production cluster, representing quote services, risk engines, onboarding APIs, and market event processing workers. During sudden spikes in load, such as market openings or news-driven requests, the control plane must process all objects within the sync window.

With low concurrency and 400 HPA objects, evaluation could be delayed by multiple 15-second cycles, leading to scaling latency of 45 to 90 seconds. This is too long for SLOs of 500ms for trading APIs, as scaling latency can violate availability SLOs.

The key architectural issue is that the autoscaler's reaction speed impacts availability SLOs, not merely infrastructure metrics. Teams setting p99 latency targets for trading APIs must now account for scaling latency in their error budgets, which many have not done. The bottleneck lies in the upstream kube-controller-manager's default --concurrent-horizontal-pod-autoscaler_syncs flag, set to 5.

This means evaluating 500 HPA objects requires at least 100 serial goroutine iterations, consuming 67% of the 15-second sync window.

AWS's July 28, 2026 announcement raises this concurrency to up to 40 times the default, reducing evaluation cycles from ~100 to ~5 for a 500-HPA-object cluster. This transformation shrinks the time from load detection to the beginning of scaling from multiple cycles to a fraction of the first, transitioning from minutes to seconds.

Crucially, EKS Provisioned Control Plane is AWS's premium offering, providing a dedicated, non-shared control plane that can handle increased concurrency without impacting other tenants due to reserved capacity on the API server.