{
  "id": 20615233,
  "title": "Our data-analyst agent lost to a single prompt: 0.55 vs 0.70 on DABench with a 1.5B model",
  "url": "https://urgent.news/2026/11/18/our-data-analyst-agent-lost-to-a-single-prompt-0-55-vs-0-70-on",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-18T15:36:24.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/protechscacademy/our-data-analyst-agent-lost-to-a-single-prompt-055-vs-070-on-dabench-with-a-15b-model-ohk"
  },
  "original_language": "en",
  "account": "A data-analyst agent built by ProTechSC Academy was outperformed by a simple one-shot baseline, with the agent scoring 0.55 vs 0.70 on DABench using a 1.5B model. The agent, which plans analysis, writes pandas code, runs it in a sandbox, and repairs its own code, failed to consistently generate correct tool calls and repair code over multiple turns. The agent's performance was hampered by three main issues: 1) the 1.5B model could not reliably execute tool calls, resulting in 18 out of 40 questions failing, 2) self-repair loops often went in circles, adding only one extra correct answer, and 3) hard questions required a single clean program rather than multiple patches, which the agent's plan-and-execute protocol struggled with. The main agent used significantly more tokens and time than the single-shot code, which was cheaper and more accurate. The study concludes that agent protocols should be chosen based on the DEV set before the evaluation run, and emphasizes the importance of measuring cost alongside accuracy.",
  "summary": "We built a data-analyst agent : it reads a question about a CSV file, plans the analysis, writes pandas code, runs it in a sandbox, reads the error if something breaks, and repairs its own code. It is the kind of loop most agent tutorials recommend. Then we compared it with the simplest possible baseline: ask the same model once for a program, run it, done. The one-shot baseline won, 0.70 vs 0.55…",
  "key_points": [],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}