We nearly published a 2x regression that didn't exist

We were on the verge of publishing a performance write-up that claimed three queries had slowed down by 2 to 4 times. However, upon closer inspection, we discovered that the disparity was due to a single factor: the same binary, running on the same machine, produced different results depending on the time of day. To investigate a change to Samyama Graph, we measured LDBC SNB Interactive query IC9 on one commit, one dataset, and one 16-core workstation, four hours apart.

The median timings were 2,822 ms in the morning and 4,912 ms in the evening, indicating a slowdown of 1.74 times. To confirm this, we ran the benchmark again with the same binary and minutes apart: the median times were 2,011 ms and 2,490 ms, a difference of 24%. We repeated the process back to back, and the results showed IC9 was 1.97 times faster, while IC10, IC11, and IC12 showed negligible differences.

This change resulted in a genuine ~2x improvement on IC9 and no regression in other queries. The 2-4x regressions were a result of the host's variability, not the code. We only caught it because a 4x regression from a small change seemed implausible and warranted a re-measurement. A smaller, real regression within the same noise band would have gone unnoticed and shipped unnoticed.

This incident highlights the importance of considering benchmark numbers' context. A regression can remain hidden within the noise if its run-to-run variance is high, such as 24%. A nightly CI gate set to trigger on a 25% threshold would flag noise some nights while missing genuine 20% regressions on others. Conversely, an improvement can be fabricated by timing alone, measuring a baseline in the morning and a new version in the evening on a drifting host.

This phenomenon is not unique to our engine, benchmark suite, or language; it is a general property of measuring wall-clock time on shared physical machines. To prevent such errors, we implemented two measures: a calibration line in every run, which times a short, CPU-bound operation at the start and end, and a rule for before/after claims, which requires both numbers to come from one back-to-back session on one machine.