The CPU limit that throttled a service using a third of its quota

A Go API experienced a p99 of 1.4 seconds, but showed only an 18 millisecond median latency. No slow queries, lock contention, or garbage collection pauses were present. The CPU dashboard indicated the pods were using roughly 35 percent of their limit consistently. However, Sergey Shinder later realized the issue was not CPU-related.

The problem stemmed from the CPU limit set at 500m, which allocated 50 milliseconds of CPU time per 100 millisecond period using the CFS quota. The service ran on 64-core nodes, so the Go runtime set GOMAXPROCS to 64 and scheduled work on multiple threads. These threads consumed the entire 50 millisecond quota in just two milliseconds of wall clock time, causing all threads to be descheduled until the next period began.

This resulted in a 98 millisecond stall for any request unlucky enough to be in flight. The throttling ratio was 41 percent of periods, which contradicted the average utilisation of 35 percent. Three changes were made to resolve the issue: setting GOMAXPROCS based on the cgroup quota using automaxprocs, increasing limits to twice the observed steady-state request, and removing CPU limits for latency-sensitive services to make them burstable.

These changes eliminated the throttling, and the CPU limit is now monitored alongside utilisation with an alert above five percent. A CPU limit is not a ceiling on usage; it is a hard stop that can be reached mid-request, and average utilisation will not reveal it.