{
  "id": 9975453,
  "title": "Tuning a Server for Benchmarking",
  "url": "https://urgent.news/2026/09/26/tuning-a-server-for-benchmarking",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-09-26T11:48:40.000Z",
  "source": {
    "name": "Lobsters",
    "slug": "lobsters",
    "url": "https://david.alvarezrosa.com/posts/tuning-a-server-for-benchmarking/"
  },
  "original_language": "en",
  "account": "Measuring code performance begins with repeatability: a 2% boost is imperceptible amidst 5% fluctuation. However, on an untuned computer, the same program can execute 2-3% faster or slower in subsequent runs. This post walks through tuning a simple benchmark step by step, measuring after each change, until the results become consistent. It's important to note that benchmark tuning differs from performance tuning: while a benchmark seeks repeatable results, a production system strives for maximum speed. Our example involves summing an array of doubles in short bursts, interspersed with brief idle periods. Each iteration runs 256 additions after a 2 ms pause, with the idle time excluded from the measurement. The pause and resume timing functions ensure the sleep period doesn't affect the measured duration, and DoNotOptimize keeps the result available after the compiler's optimization. Compile the code in release mode with maximum optimizations (-O3), native CPU architecture (-march=native), native CPU tuning (-mtune=native), and enable Loop Transformation Technology (-flto) and aggressive floating-point optimizations (-ffast-math). Run ten repetitions and average the results. The key metric is the coefficient of variation (cv), calculated as the standard deviation divided by the mean. Approximately 3% of variation in run-to-run performance indicates that any improvement below this threshold is negligible. Before adjusting any parameters, examine the current state. Tools like lstopo provide a comprehensive view of the machine, including cache levels, cores, simultaneous multi-threading (SMT) pairs, and PCIe devices connected to them. On a single-socket laptop, starting with CPU 4 yields performance from an E-core at lower clock speeds, while CPU 12 might lack the L3 cache. In a homelab server, all cores are equivalent, making it an ideal benchmarking environment. PCIe configuration becomes crucial when the benchmark involves I/O, as it determines which NVMe or NIC is being tested and which NUMA node the device connects to. The operating system's scheduler can migrate the benchmark between cores, causing warm caches to dissipate. Moreover, on hybrid CPUs, performance and efficiency cores operate at significantly different speeds, resulting in bimodal performance. Pinning the benchmark to a single core (preferably a P-core on hybrid CPUs) reduces the mean runtime to 55.3 microseconds, and the coefficient of variation improves to less than 1.06%. Pinning the benchmark to a single core ensures that the core's clock never drops between bursts, maximizing its performance. Further optimization involves reserving the core exclusively for the benchmark, either through kernel command line options (isolcpus=2 nohz_full=2 rcu_nocbs=2) or by creating a cgroup with cpuset. By default, Linux adjusts the CPU frequency based on load, causing the benchmark to start on a cold clock and end on a hot one. Switching the frequency governor to performance keeps the clock speed high throughout the benchmark, resulting in a mean runtime of 54.9 microseconds and a coefficient of variation of 0.79%. Since the benchmark never wakes up on a cold clock, performance remains consistently high. However, the benchmark still shares execution units and L1/L2 caches with its SMT sibling. Disabling SMT entirely further reduces the coefficient of variation to 0.26%, tripling the improvement. The core now operates with exclusive execution units and caches, resulting in even more stable performance. Even with the performance governor, turbo frequencies can vary with temperature and power budget, leading to lower clock speeds on a warm machine compared to a cool one. Disabling turbo enables stable clocks, although this may result in lower peak performance since the benchmark prioritizes relative performance improvements over absolute speed. In low-latency production environments, maintaining turbo frequencies is crucial for optimizing every nanosecond of performance. Some high-frequency trading firms go even further by running overclocked servers at a fixed all-core frequency above stock, trading off peak performance for stability and lower latency. The table below summarizes the journey from a 3% noisy benchmark to a 0.26% coefficient of variation, achieving a three-fold improvement in performance. Each row adds a new optimization on top of the previous ones, demonstrating the significant impact of each change. Some additional tweaks, such as disabling address space layout randomization, the NMI watchdog, or transparent huge pages, can further enhance performance on busier machines. The bench-remote.sh script automates applying all these optimizations, but they only last until the next reboot, which is ideal for benchmarking purposes. For more information, consult the mailing list at david@alvarezrosa.com.",
  "summary": null,
  "key_points": [
    "Compile code in release mode with maximum optimizations",
    "Pin benchmark to single core (prefer P-core on hybrid CPUs)",
    "Disable SMT to achieve 0.26% coefficient of variation"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}