Urgent.News

What's breaking now, across thousands of outlets.

Tech

Our data-analyst agent lost to a single prompt: 0.55 vs 0.70 on DABench with a 1.5B model

We built a data-analyst agent : it reads a question about a CSV file, plans the analysis, writes pandas code, runs it in a sandbox, reads the error if something breaks, and repairs its own code. It is the kind of loop most agent tutorials recommend. Then we compared it with the simplest possible baseline: ask the same model once for a program, run it, done. The one-shot baseline won, 0.70 vs 0.55…

A data-analyst agent built by ProTechSC Academy was outperformed by a simple one-shot baseline, with the agent scoring 0.55 vs 0.70 on DABench using a 1.5B model. The agent, which plans analysis, writes pandas code, runs it in a sandbox, and repairs its own code, failed to consistently generate correct tool calls and repair code over multiple turns.

The agent's performance was hampered by three main issues: 1) the 1.5B model could not reliably execute tool calls, resulting in 18 out of 40 questions failing, 2) self-repair loops often went in circles, adding only one extra correct answer, and 3) hard questions required a single clean program rather than multiple patches, which the agent's plan-and-execute protocol struggled with.

The main agent used significantly more tokens and time than the single-shot code, which was cheaper and more accurate. The study concludes that agent protocols should be chosen based on the DEV set before the evaluation run, and emphasizes the importance of measuring cost alongside accuracy.

Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.

Read the original at dev.to →

More in Tech

More from Wednesday 18 November →