Overreaching causal language in the social sciences

Across the social sciences, numerous studies utilize cross-sectional designs that uncover associations but are typically incapable of substantiating direct causal claims. However, authors of such papers may assert or insinuate causal claims nonetheless. In order to ascertain the frequency of this "overreaching" causal language, a study analyzed 194,631 cross-sectional articles employing large language models.

Over the timeframe of 1980 to 2024, an average of 46% of these articles contained causal language within their titles or abstracts, with the annual rate surging nearly threefold since the year 2000 from 20% to 60%. To investigate the impact of such language, a human-subjects experiment with N = 1,105 participants was conducted, revealing that readers often perceive abstracts featuring this phrasing as providing causal evidence.

Nonetheless, methodological labels (β = -0.4, 95% confidence interval -0.56 to -0.19) and associational wording (β = -0.3, 95% confidence interval -0.43 to -0.07) tended to diminish this inclination. Experiments involving five different large language models (LLMs) demonstrated that model summaries of these articles (N = 100 each) could exacerbate the overstatement of causality, eliminating cautionary language and introducing causal claims where only associational phrasing was utilized; however, prompting caution did mitigate this pattern.

This finding is derived from a recent paper by Calvin Isch, Timothy Dörr, Neil Fasching, Grace Jennings, and Duncan J. Watts.