AIs vs. humans in economics papers
Is agentic research any good? Dr Yanagizawa-Drott and his colleague, Olaf Willner, used an AI verifier to check 1,000 APE papers. None was error-free; 842 contained at least one severe error, including ones invalidating headline results. To see roughly how Ape stacks up against human primates, The Economist teamed up with Refine, the research-verification firm […] The post AIs vs. humans in…
A recent study examined the accuracy of AI-generated research compared to human-authored papers in the field of economics. Researchers Yanagizawa-Drott and Willner utilized an AI verifier to assess 1,000 such papers, revealing that none were entirely error-free. In fact, 842 contained at least one severe error, including instances that invalidated the main findings of the research.
To better gauge AI's performance against human economists, The Economist collaborated with Refine, a research-verification firm established by Dr. Golub. The team randomly selected 100 papers from the most recent nine months of top-tier journals, including QJE and JPE, and subjected them to rigorous scrutiny using advanced AI models alongside manual error checking by a human reviewer.
This analysis highlighted 4,158 issues across the 100 papers, with one in ten papers exhibiting at least one grave error. Other problems were attributed to misinterpretations or inconsistencies between tables and text.
While human economists exhibited a lower error rate, AI models showed promise in certain aspects. An experiment conducted by researcher Alexis Akira Toda from Emory University found that AI could identify candidate flaws in economic theory papers more efficiently, although it did not entirely replace human judgment in pinpointing these flaws.
Another study comparing AI and human-authored papers discovered that the primary differences lay in ideas rather than execution. Michele Zampa of the Geneva Graduate Institute utilized an AI referee to evaluate 981 AI-generated papers alongside 41 human ones, focusing on factors such as the novelty of research questions and acknowledgment of methodological limitations.
AI performed better in fields with extensive existing research, while human reviewers maintained their accuracy regardless of the literature size.
In a separate endeavor, Swiss researchers from ETH Zürich allowed AI to reproduce 48 social-science papers using only raw data and study descriptions. The discrepancies between the original and AI-generated versions were attributed to an approximately equal balance between AI and human errors. However, as AI models improved, the advantage shifted in their favor.
This story, originally published by The Economist, provides a comprehensive look into the ongoing competition between AI and human economists in the realm of economics research.
Written by urgent.news from Marginal Revolution's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.