'No difference' results may be wrong conclusion from medical data research, scientists warn

Researchers from the Universities of Manchester, Oxford, and Arkansas are cautioning scientists against treating non-significant results as evidence that no effect exists. The issue lies in the common statistical mistake of interpreting a p-value greater than 0.05 as definitive proof of the absence of an effect. However, this misinterpretation is prevalent in approximately half of research papers and conference presentations.

Contrary to popular belief, a non-significant result simply indicates insufficient evidence to conclude a difference exists, not proof that a difference does not exist. The same statistical result can arise either due to a genuine lack of meaningful effect or because of small sample sizes or highly variable data. Ignoring this distinction can lead to oversimplification of scientific findings, potentially overlooking important effects.

The team argues that failing to recognize this distinction risks misrepresenting the significance of findings and may hinder the detection of genuine risks or promising therapies. To address this problem, the researchers advocate for the use of equivalence testing, a statistical approach that focuses on whether any effect that exists is too small to matter in practical terms.

The two one-sided tests procedure (TOST), a widely adopted method in psychology, medicine, and pharmaceutical regulation, is proposed as a means to distinguish between genuinely negligible effects and inconclusive results due to insufficient evidence. The authors emphasize that equivalence testing can enhance the quality of scientific reporting and prevent nonsignificant findings from being mistakenly interpreted as evidence of no effect.

Co-authors David Eisner and Jakub Tomek highlight the importance of this approach in providing a clearer understanding of the data and improving the confidence in reported conclusions.