That 98% is a model score, not a rate

A circulating claim this week suggested that 98% of truck drivers engage in dangerous driving practices. To verify this assertion, I delved into the source material. The source material was a PLOS ONE paper (20(8):e0328899), built upon the Kaggle DBRA24 dataset, containing 120,000 trip records, 26 features, collected in California during January 2023 and released under the CC0 license.

Contrary to the claim, the 98% figure does not represent a prevalence rate. Instead, it signifies the model's accuracy in distinguishing anomalous driving rows from normal ones within that dataset. Accuracy is a narrow metric that answers the question: how often does the classifier's predictions align with the labels it was trained and tested on? However, it does not provide insight into the actual prevalence of dangerous driving behavior among truck drivers.

The issue becomes more pronounced when dealing with imbalanced classes. For instance, if only 5% of the rows carry an anomalous label, a model that consistently predicts normal behavior would achieve an impressive 95% accuracy, while failing to identify any actual anomalies. Therefore, a headline stating "98% of truck drivers drive dangerously" merely indicates the model's performance on the given dataset, but it offers no information about the actual prevalence of such behavior in the trucking population.

Additionally, there is no evidence in the paper to support the claim that five rows were labeled at random. As a general rule, when a model score is presented as a factual statement about the world, it is crucial to ask two questions: what does the score actually measure, and what is the denominator used in calculating that score? Accuracy, precision, recall, and AUC are four distinct metrics, each providing different insights. Only one of these metrics—accuracy—can be considered a reliable representation of the world.