He Wants AI Benchmarking To Be a Science, Not a Dark Art
Moritz Hardt, author of a new book on benchmarks, believes these “standardized tests” for artificial intelligence are fundamentally broken — but not unfixable. The post He Wants AI Benchmarking To Be a Science, Not a Dark Art first appeared on Quanta Magazine
Benchmarks are essential in AI, serving as both measuring tools and influence over the entire field. In 1987, the Defense Advanced Research Projects Agency's benchmark on automatic speech-to-text translation revealed that AI could not surpass statistical methods, leading to an AI winter. However, ImageNet in 2012 reignited interest, launching the "deep learning revolution."
Nowadays, benchmarks for large language models, like OpenAI's GPT-6 Astra, are created and solved regularly, aiming to measure various aspects of AI, including domain-specific expertise and "general intelligence." However, many in the field believe benchmarking has become flawed and seek a new approach.
Moritz Hardt, a computer scientist and director at the Max Planck Institute for Intelligent Systems, contends that benchmarking has been flawed for decades. In his 2026 book, "The Emerging Science of Machine Learning Benchmarks," Hardt argues that benchmarks have been a mistake, driving progress unintentionally. He suggests that instead of creating a replicability crisis, benchmarks have mainly driven progress "by sheer luck."
This article delves into the origins of benchmarks and their evolution into the flawed system we have today, exploring why machine learning has managed to avoid a scientific crisis and how benchmarks might regain their credibility.
Written by urgent.news from Quanta Magazine's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
