How We Stabilized a Bazel CI Runner by Fixing Resource Limits and Container Cleanup

A monorepo pipeline running Go, Node.js, and C++ code was experiencing slow build times and occasional failures. The Bazel build system was being OOM-killed during peak hours, causing long retry times and reduced frequency of pushes. Engineers spent days investigating the root cause before discovering the issue lay in resource limits and container cleanup on the CI runner host.

The runner, with 6 cores and 12 threads, was showing a load average of 32.7 and swapping space was being used. The CI configuration block stated the runner could run a maximum of two Bazel containers at once, with each build consuming 5 CPU threads and 24 GiB of memory. However, the runner was actually running 9 jobs simultaneously, each requesting more resources than the machine could provide.

To stabilize the CI runner, the first step was to impose a hard cgroup limit, allowing three jobs to run simultaneously with four CPUs and 14 GiB of capacity each. Next, four jobs were run with four CPUs and 12 GiB each, followed by five jobs with three CPUs and 11 GiB each, at which point the system stabilized. The .bazelrc file was then gradually adjusted to limit CPU to 3 threads (5 x 3 = 15, 1.25x SMT) and memory to 9 GiB under the original 11 GiB cap.

Two rules presented unique challenges. The value of local_resources must be strictly below the cgroup's upper limit, as resource-intensive processes would otherwise be queued instead of being OOM-killed. Memory values should be determined based on actual measured resident memory, not the 11.37 GiB working set, as the difference between these values represents reclaimable page cache that the kernel can free when needed.

Additionally, some jobs were moved from the overloaded runner to a lighter runner to free up build slots. The integration tests' helper containers were leaking due to improper cleanup processes, as they were not labeled with Testcontainers and were therefore not removed by the host's garbage collector. These containers ran for months, consuming outdated resources.

To address the leak, the label ITEST_CONTAINER_LABELS was added to each calling site, ensuring the host's garbage collector would remove these containers after they completed. With these adjustments, the Bazel CI runner was successfully stabilized, demonstrating the importance of following resource limits and properly managing container cleanup on shared hosts.