{
  "id": 8400513,
  "title": "Efficiency Hallucination: Every Model Rewrote Code That Couldn't Get Faster",
  "url": "https://urgent.news/2026/09/19/efficiency-hallucination-every-model-rewrote-code-that-couldnt-get",
  "topic": "ai",
  "section": "AI",
  "published": "2026-09-19T06:41:08.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/abyzgenic/efficiency-hallucination-every-model-rewrote-code-that-couldnt-get-faster-182c"
  },
  "original_language": "en",
  "account": "A Columbia University study published on arXiv in September reveals a phenomenon known as \"Efficiency Hallucination\" in large language models (LLMs) when asked to optimize code. Researchers Sarah Wilson, Gail Kaiser, and Patrick Musau tested nine models from Claude, GPT, and Gemini families on five problems from EffiBench. They paired the top-performing human solutions with deliberately degraded versions and asked the models to optimize both. Under a standard prompt, all nine models rewrote the already optimal code without fail. The researchers dubbed this behavior \"the Evaluation Trap,\" as every benchmark a model has been trained on rewards producing an edit, even if it's unnecessary. To combat this, the authors introduced a simple intervention: instruct the models to only suggest an edit if they are 90% confident it improves execution speed; otherwise, they should output \"ALREADY_OPTIMAL.\" This resulted in a significant decrease in over-editing on optimal code, from 100% to 55.6%. However, the models still overestimated their confidence level, reporting 90% even when they were unsure, which is a common overconfidence problem. The researchers note that the true solution lies in actual performance testing rather than just code optimization. They also found a \"Capability-Calibration Inversion,\" where better-performing models were often worse at recognizing when no optimization is needed. The study highlights the importance of evaluating not just the models' ability to optimize code but also their calibration in determining when a code snippet is already optimal.",
  "summary": "Okay, this is going to sound dumb, but I spent Sunday evening asking three different models to make a function faster that could not be made faster, and every single one of them did it anyway. Confidently. With a comment at the top explaining the speedup. The function was a two-pointer sweep over a sorted list. It's O(n), it touches each element once, and I had already benchmarked it. I pasted it…",
  "key_points": [
    "\"Efficiency Hallucination\" phenomenon in LLMs optimizes already optimal code",
    "\"Evaluation Trap\" causes models to rewrite optimal code despite no improvement",
    "Introducing confidence threshold reduces over-editing from 100% to 55.6%"
  ],
  "editors_take": "The study's findings highlight that large language models' tendency to rewrite already optimal code undermines their usefulness, and addressing this requires evaluating not just optimization capabilities but also calibration of confidence levels.",
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}