A 40ms Go garbage collector pause caused by swap

A Go garbage collector pause was caused by swap due to metadata being stored in swap, resulting in a 40ms pause. The researcher ran swap in production to handle memory spikes, but found that the pause was longer than expected. The garbage collector reads metadata outside the heap in a stop-the-world pause, which can be stored in swap.

By conducting experiments on a Hetzner box with kernel 6.8 and MGLRU enabled, the worst pause was found to be 40ms, with 39 of those 40ms spent on 228 page faults. The page faults occurred during the GC's bookkeeping process, as the kernel evicts pages by age and sends least recently accessed pages to swap. This stop-the-world pause can cause other goroutines to stop and wait, leading to significant delays.

Although building a 511 KiB message takes a few milliseconds, it can increase to 105 ms or even 903 ms on Hetzner’s network volume due to the metadata pause. While swap isn't necessarily evil, it doesn't perform well with garbage collection, and in production, the researcher found it costly. They discovered that Go 1.26's Green Tea garbage collector has negligible impact on the metadata pause.