{
  "id": 20937628,
  "title": "The most misunderstood graph in AI just got a major update",
  "url": "https://urgent.news/2026/11/20/the-most-misunderstood-graph-in-ai-just-got-a-major-update",
  "topic": "ai",
  "section": "AI",
  "published": "2026-11-20T18:15:15.000Z",
  "source": {
    "name": "MIT Technology Review",
    "slug": "mit-technology-review",
    "url": "https://www.technologyreview.com/2026/11/20/1149399/the-most-misunderstood-graph-in-ai-just-got-a-major-update/"
  },
  "original_language": "en",
  "account": "A new report from the AI evaluation nonprofit METR reveals that certain AI capabilities continue to advance exponentially, but they lag behind in tasks requiring judgment compared to coding proficiency. METR evaluated models by assessing their performance in a variety of tasks, some previously completed by humans. The difficulty of each task was measured by the time taken by human testers, and models were rated based on their ability to complete tasks of increasing complexity. Every four months, the length of tasks that frontier models could successfully complete doubled, indicating rapid improvement. However, recent updates to the time horizon plot have added more complex tasks and those that are more open-ended. A key question in AI is whether progress on verifiable tasks (with automatic success or failure) will transfer to tasks with less clear success criteria. While AI systems excel at coding and math, where automatic verification is possible, they struggle with tasks like selecting research directions, where success is less defined. METR's previous plot did not include such tasks due to their difficulty in evaluation. However, the updated report includes more subjective tasks, such as designing new AI evaluations for METR to use and scoring philosophical arguments. Despite AI's ability to outperform humans in software engineering, it fails to create useful evaluations in these tasks. Experts view the incorporation of new evaluation types as a positive step, as AI evaluations need to become more creative. Researchers remain divided on the interpretation of performance differences between easily verifiable and hard-to-verify tasks, but METR's latest report indicates that models may be improving in the latter area, albeit still far from human performance.",
  "summary": "A new report from the AI evaluation nonprofit METR finds that some AI capabilities are continuing to improve along an exponential trajectory—but it also indicates that models aren’t quite as good at tasks that require judgment as they are at coding. The report is an update to the organization’s time horizon plot, which was widely…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}