7 checks I run on a number before it goes in front of money
The number that embarrasses you is never the obvious lie. It's the ordinary-looking one everybody nods past, because it's already in the deck and the deadline was yesterday. I've spent weeks checking claims for people. The same seven questions catch most of it. Run them on your own numbers before someone else does. 1. Is it a rate or a score? "98% accurate" is usually a model score, not a rate of…
Before a number is presented with money on the line, it should undergo seven critical checks. The first check involves determining if the figure is a rate or a score. For instance, claiming a 98% accuracy is likely a model score, not an actual rate of anything tangible. A score indicates a claim about a dataset, whereas an accuracy rate speaks to the world at large.
The second step is identifying the denominator. For example, stating that something is 40% faster than another entity requires specifying what it is faster than and how it was measured. A number devoid of its denominator is essentially a feeling with a percentage sign attached. Third, one must ascertain who selected the sample. If the chosen participants volunteered to participate, the sample may have skewed the results.
It is crucial to inquire who was excluded, as their absence may be the crux of the story. Fourthly, establish whether the value is a maximum, a mean, or a median. A single exceptional case does not represent the norm. If removing the best case alters the headline, the figure is not a headline but a best case scenario. Fifthly, ensure the version and units align with the claim.
The benchmark cited should correspond to the version used, and the hardware employed. Often, the units cited are inaccurate, such as using Rs instead of $, milliseconds instead of seconds, per-user versus total, or monthly versus annual. The sixth step requires confirming that the figure remains valid under context. For instance, being 10% better than the baseline is only meaningful if the baseline is what users actually utilize.
Lastly, ascertain who else must be correct for the claim to hold true. Each number is contingent upon someone else's dataset, API, or report. If the source cannot be traced, the claim is not independently verified and carries inherent risks.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
