Zero Cost Is Not Zero Variance: Four Myths About Repeating an AI Coding Run

Claim 1: The same prompt does not always lead to identical results

Run the same task multiple times against the same codebase, and it is unlikely you will obtain eight identical differences. Factors such as sampling methods, tool call ordering, network retries, and the sequence in which files are read can shift the run in slightly different directions. Therefore, a single successful run is merely an anecdote about one sample, not a characteristic of the tier.

To better understand the underlying distribution, write it down. This simple act takes less effort than debating the outcome in a pull request. Once you have rows, the conversation shifts from "it worked for me" to "median acceptable, tail unacceptable."

Claim 2: A green run does not guarantee a flawless workflow

A passing test suite indicates that one sample passed, but it offers no insight into the likelihood that subsequent samples will pass. The problem lies in how we remember the information. We tend to store the agent's diff and test outcome together in our minds, retaining only the successful version. To address this, record both columns for every run.

The corrected model separates process from task. An exit code of zero confirms that the command finished, a test result indicates that the change was acceptable, and a diff count reveals how much review surface the change created. Frequently, these three signals diverge enough that condensing them into a single boolean value can mislead you.

Claim 3: The median is the most relevant number, not the average

Medians are intuitive and tails are costly, and free tiers often generate tails. If seven runs complete in a few minutes and the eighth takes an hour, the median appears healthy while the afternoon is wasted. Anyone who has experienced a stuck job knows that the average can ruin your day. Instead of focusing on the average, compute the tail ratio: the worst run divided by the median run.

This ratio, derived from the same ledger rows, helps you determine whether a tool with a stable median and a large tail ratio is suitable for exploration or unsuitable for time-sensitive tasks. Before making a decision, calculate both numbers and consider the type of tool you are dealing with.

Ledger example (8 runs, 4 columns):

Run #, Time, Exit Code, Added Files, Deleted Files

1, 0.5s, 0, 2, 1

2, 0.6s, 0, 1, 2

...

8, 1.2s, 1, 3, 4

Summarize the data using a short awk script, focusing on the median time, minimum and maximum times, and the tail ratio. Add the task verdict yourself to ensure accuracy, as the wall time cannot determine whether a diff is correct. Include a second column for the exit status of your test command to maintain honesty in the ledger when a fast run produces an incorrect patch.

Claim 4: Variance stems from the environment, not the model itself

Variance usually originates from the environment surrounding the model. Common sources include shared shells with warm package caches, different locales formatting numbers, and proxies silently retrying requests. These factors have nothing to do with the model's weights. When you change the environment between runs, you are measuring the environment, not the agent.

Treat the agent as one variable among several, and hold the others fixed or record them. A warm cache and a different locale are environmental variables, which explain why two teams may obtain different results. By acknowledging the role of the environment, you can more accurately assess the true variance of the model.