{
  "id": 2970893,
  "title": "We nearly published a 2x regression that didn't exist",
  "url": "https://urgent.news/2026/08/24/we-nearly-published-a-2x-regression-that-didnt-exist",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-08-24T07:02:47.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/samyama-ai/we-nearly-published-a-2x-regression-that-didnt-exist-3bii"
  },
  "original_language": "en",
  "account": "We were on the verge of publishing a performance write-up that claimed three queries had slowed down by 2 to 4 times. However, upon closer inspection, we discovered that the disparity was due to a single factor: the same binary, running on the same machine, produced different results depending on the time of day. To investigate a change to Samyama Graph, we measured LDBC SNB Interactive query IC9 on one commit, one dataset, and one 16-core workstation, four hours apart. The median timings were 2,822 ms in the morning and 4,912 ms in the evening, indicating a slowdown of 1.74 times. To confirm this, we ran the benchmark again with the same binary and minutes apart: the median times were 2,011 ms and 2,490 ms, a difference of 24%. We repeated the process back to back, and the results showed IC9 was 1.97 times faster, while IC10, IC11, and IC12 showed negligible differences. This change resulted in a genuine ~2x improvement on IC9 and no regression in other queries. The 2-4x regressions were a result of the host's variability, not the code. We only caught it because a 4x regression from a small change seemed implausible and warranted a re-measurement. A smaller, real regression within the same noise band would have gone unnoticed and shipped unnoticed. This incident highlights the importance of considering benchmark numbers' context. A regression can remain hidden within the noise if its run-to-run variance is high, such as 24%. A nightly CI gate set to trigger on a 25% threshold would flag noise some nights while missing genuine 20% regressions on others. Conversely, an improvement can be fabricated by timing alone, measuring a baseline in the morning and a new version in the evening on a drifting host. This phenomenon is not unique to our engine, benchmark suite, or language; it is a general property of measuring wall-clock time on shared physical machines. To prevent such errors, we implemented two measures: a calibration line in every run, which times a short, CPU-bound operation at the start and end, and a rule for before/after claims, which requires both numbers to come from one back-to-back session on one machine.",
  "summary": "We were about to ship a performance write-up claiming three queries had gotten 2-4x slower. All three numbers were wrong, and the reason is one every benchmark author should worry about: the same binary , on the same machine , running the same query , gave different answers depending on what time of day we ran it. The measurement that didn't add up While checking a change to Samyama Graph , we…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}