{
  "id": 14962593,
  "title": "He Wants AI Benchmarking To Be a Science, Not a Dark Art",
  "url": "https://urgent.news/2026/10/16/he-wants-ai-benchmarking-to-be-a-science-not-a-dark-art",
  "topic": "ai",
  "section": "AI",
  "published": "2026-10-16T13:25:08.000Z",
  "source": {
    "name": "Quanta Magazine",
    "slug": "quanta-magazine",
    "url": "https://www.quantamagazine.org/he-wants-ai-benchmarking-to-be-a-science-not-a-dark-art-20261016/"
  },
  "original_language": "en",
  "account": "Benchmarks are essential in AI, serving as both measuring tools and influence over the entire field. In 1987, the Defense Advanced Research Projects Agency's benchmark on automatic speech-to-text translation revealed that AI could not surpass statistical methods, leading to an AI winter. However, ImageNet in 2012 reignited interest, launching the \"deep learning revolution.\" Nowadays, benchmarks for large language models, like OpenAI's GPT-6 Astra, are created and solved regularly, aiming to measure various aspects of AI, including domain-specific expertise and \"general intelligence.\" However, many in the field believe benchmarking has become flawed and seek a new approach.\n\nMoritz Hardt, a computer scientist and director at the Max Planck Institute for Intelligent Systems, contends that benchmarking has been flawed for decades. In his 2026 book, \"The Emerging Science of Machine Learning Benchmarks,\" Hardt argues that benchmarks have been a mistake, driving progress unintentionally. He suggests that instead of creating a replicability crisis, benchmarks have mainly driven progress \"by sheer luck.\" This article delves into the origins of benchmarks and their evolution into the flawed system we have today, exploring why machine learning has managed to avoid a scientific crisis and how benchmarks might regain their credibility.",
  "summary": "Moritz Hardt, author of a new book on benchmarks, believes these “standardized tests” for artificial intelligence are fundamentally broken — but not unfixable. The post He Wants AI Benchmarking To Be a Science, Not a Dark Art first appeared on Quanta Magazine",
  "key_points": [
    "Benchmarks are crucial in AI, measuring tools and field influence.",
    "Moritz Hardt criticizes benchmarks as flawed for decades.",
    "New approach sought to restore benchmarks' credibility."
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}