Experimentally validated digital twins
There is a new paper by Raj Chetty, Jamie Fogel, Lawrence F. Katz, Kadeem Noray, Sonya Porter, and James Reisinger: Modern program evaluation prioritizes evidence from randomized controlled trials (RCTs) because prior research found that observational estimates of program impacts often diverge substantially from experimental benchmarks. We revisit this finding — which was based on […] The post…
A groundbreaking paper by Raj Chetty, Jamie Fogel, Lawrence F. Katz, Kadeem Noray, Sonya Porter, and James Reisinger explores the potential of digital twins in program evaluation. Traditionally, program impact has been assessed through randomized controlled trials (RCTs), but observational estimates have often fallen short of experimental benchmarks. This study revisits that notion using large-scale administrative data, specifically anonymized income tax records spanning 25 years and covering all American adults.
By leveraging machine learning models, the researchers were able to predict counterfactuals, or hypothetical scenarios, from rich data on prior work histories, demographics, and residential locations. Instead of testing whether these observational estimates are unbiased, the study utilized 37 RCTs to estimate the distribution of their bias. The findings revealed that machine learning estimates closely align with RCT estimates, exhibiting a root mean squared error (RMSE) comparable to that of the RCTs themselves.
The approach employed in this study significantly outperformed traditional parametric estimators, reducing RMSE by 60%. This improvement is attributed to larger sample sizes and more comprehensive covariates, enabling the use of more sophisticated, higher-dimensional models for constructing counterfactual scenarios. Applying this method to additional sites and enrollment years beyond the experimental sample, the researchers discovered that treatment effects on earnings vary considerably across locations and over time, even within the same program.
Programs that successfully place more participants into jobs within the target sector yield the most substantial earnings gains. Furthermore, these positive impacts persist even after participants transition out of their initial jobs. These insights offer valuable guidance for enhancing job training programs and demonstrate a scalable approach to program evaluation that harnesses observational data to create experimentally validated "digital twins."
This revolutionary approach promises to transform economic methodologies, as highlighted by the authors.
Written by urgent.news from Marginal Revolution's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
