Seu agente não é consistente: por que uma única execução não prova que ele funciona

The same agent, the same model, the same prompt, and the same environment were used, but running the same experiment again produced different results. The paper Identical Runs, Different Results: Benchmarking AI Coding Agents on Open-Weight Models (arXiv:2609.33812) conducted six agent-model combinations, each repeated 52 times under fixed configurations - data, prompt, budget, machine, and parallelism, for a total of 584 executions.

The variation between identical runs of the same combination was greater than the difference between different combinations. Rankings inverted, with what seemed best in one round falling in another. Comparing with few executions does not reliably order results. To distinguish small differences between agents in the study, the authors estimate tens to hundreds of runs per arm.

An improvement of about 2 percent points in pass@1 in cited works may require around 9 independent runs; 1 percent, around 36. A single demonstration or a single successful PR does not measure reliability; it measures a sample. Another operational detail is that less than one in twenty runs violated task data rules, and these runs concentrated the highest scores.

Rejecting violations and taking the best compatible result with few attempts enhanced the delivered artifact, even when few runs were not enough to rank agents with precision. Cost also diverged: in the same model, two agents came within twenty times of each other in list price, largely due to prompt cache behavior. Agent and model need to be evaluated as a pair, with repetition and compliance reported alongside quality.

In team practice, this changes the question from "Did the agent solve it once?" to "How many attempts, under what rules, and with what stability does the result repeat?" Releasing to production based on a demo, a single benchmark, or a lucky diff sort is confusing success in isolation with a controllable process. Deciding purchase, usage policy, or merge criteria needs evidence of repetition - not just a result that happened on that day.

For a deeper dive with examples and a complete architecture analysis, consult the book Engenharia de Software Assistida por IA (available on Amazon). To explore the topic more structured, related to agent governance, refer to the series Engenharia de Software Assistida por IA clicking the image below: 30% OFF coupon LEIA30. Also available on Kindle Unlimited.

References: arXiv:2609.33812 Identical Runs, Different Results: Benchmarking AI Coding Agents on Open-Weight Models (Ariño de la Rubia & Pafka, set/2026); 584 runs; 6 pairings x 52; intra-par variation - difference between pairs.