Are RCTs underpowered?

A new study examines the reliability of randomized controlled trials (RCTs) in economic research and how resources should be allocated to replicate them. Analyzing 745 key hypothesis tests from 190 articles in top economics journals, researchers discovered that the actual t-statistics frequently fall far below the ideal benchmark of 2.8, which would be expected with an 80 percent power level.

Nonetheless, the statistics still exceeded the values linked to p-hacking. This analysis indicates that many RCTs might be underpowered. By developing an empirical Bayes mixture model that takes into account selective publication, the research team estimated that roughly 30 percent of published rejections are actually false discoveries, with an average directional power of just 40 percent.

At the standard significance levels, the posterior probability of a false discovery climbs to 66 percent.

When the sample sizes are doubled, the average false discovery rate drops by five percentage points. However, tightening the significance threshold to 0.005 results in a larger reduction in false discoveries, but at the expense of fewer total discoveries. The study also outlines an optimal replication strategy for funders with a fixed budget and a specific evidence standard.

Under typical conditions, replications should focus on an intermediate range of original test statistics, with the best replication sample size being single-peaked in relation to the original t-statistic. Based on these findings, implementing successful replications could bring down the overall false discovery rate to between 1.8 and 7.8 percent, depending on the chosen evidence standard.