The Generalizability Problem, the Natural Field Experiments Answer
There’s a standard problem in the social sciences in moving from the results of a smaller-scale study to larger-scale applicability. Consider a program in a certain school district that seems to increase academic performance and high school graduation rates. Would the program work in another school district? Will it work statewide? Would it work nationwide? … Continue reading The Generalizability…
Moving from smaller-scale studies to larger applicability is a common challenge in the social sciences. A program that increases academic performance and graduation rates in one school district might not work in another. The question of how generalizable the results are lacks a simple answer. In his essay "Make science more reliable: study people as they go about their lives" (Nature, June 25, 2026, 654, pp. 863-866), John A. List presents a different perspective on this issue.
List cites the example of Scared Straight, a program run by over 30 US states between 1978 and 2015. Initially, the program showed success with 80–90% of teenage participants avoiding trouble. However, when scaled up, it failed to produce the same results. The problem lies in the fact that people, as well as societies, economies, cultures, and subcultures, are diverse. Additionally, when individuals know they are participating in a study, they may behave differently than they would if they were not being observed.
List is a proponent of "natural field experiments." These are studies designed in a way that participants are unaware they are being studied. A well-known example is the "audit study," where researchers send assistants with similar credentials but differing races or ethnicities to rent an apartment, get a bank loan, buy a car, or apply for a job.
The potential landlords, bank officials, car sellers, and employers remain unaware of the study. List has conducted numerous such studies, emphasizing that they should be pre-approved by an ethics board and should not put participants under undue stress or threat.
List's natural field experiments span various areas, such as the impact of different charitable donation announcements or matching rates in fundraising drives. He has also conducted experiments on how people react when wallets are dropped in various locations. Currently, List serves as the chief economist at Walmart, where his team conducts natural field experiments with more than 6,000 suppliers to determine factors that effectively incentivize suppliers to reduce their carbon emissions.
Written by urgent.news from Conversable Economist's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
