Evidence Levels: What a Single JSON Can and Cannot Claim
Why this lesson exists Teams often shout "this runs 10 % faster" after a single csperf run. The claim is true only if the evidence is strong enough to survive noise, variance, and the temptation to cherry‑pick. Recap — where we are in the series Ep 1: single‑shot timings mislead. Ep 2: warm‑up and repeat are essential. Ep 3: screenshots are not artifacts. Ep 4: metadata is needed for…
A single JSON file produced by csperf may appear to be a definitive benchmark, but it can be misleading without proper context. Reading only the mean of the metrics can lead to incorrect conclusions about performance, ignoring important factors like min/max values, standard deviation, and the fact that the run may have been a warm-up.
The csperf tool attaches evidence levels to each metric, indicating how statistically sound, machine-aware, or comparable the results are. The raw evidence level represents the raw number from the profiler, while sanitized evidence includes outliers removal and min/max reporting. The validated evidence level goes a step further by cross-checking the results against metadata and noise models before publishing.
This article explains the concept of evidence levels and provides a step-by-step guide on how to interpret csperf JSON artifacts. It emphasizes the importance of checking the evidence level, understanding the underlying data, and not making assumptions based solely on the mean value. The next episode will demonstrate how to compare two runs without getting bogged down in spreadsheets and chaos.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
