A Two-Model Bake-Off on Your Own Repo: Isolating Runs With Git Worktrees
Most model comparisons I read have the same flaw: the models never touch the same problem under the same conditions. One gets a carefully groomed prompt, the other gets a sloppy one. One runs against a clean checkout, the other inherits half-finished edits from the previous attempt. The conclusion tells you more about the harness than the models. I wanted a comparison I could actually trust for…
When comparing coding models, a common issue arises: they are not tested under the same conditions. One model gets a meticulously crafted prompt while another receives a more haphazard one. One runs against a clean repository, while the other inherits unfinished edits from previous attempts. This results in conclusions that reflect more about the testing setup than the models themselves.
To address this, the author created a bake-off harness using git worktrees. The harness runs two coding models against the same task in isolated checkouts, producing a table with three key metrics: whether the tests passed, the size of the diff, and whether any files were touched that shouldn't have been. This method reduces bias and provides a more trustworthy comparison.
The harness is written in bash and consists of a script called bakeoff.sh that takes three arguments: the task file, the command for model A, and the command for model B. It creates isolated worktrees for each model, runs the tests before allowing the model to make changes, and then reports the results. The key differences from using branches or clones are in the isolation and cleanup, with worktrees providing a low contamination risk and easy cleanup with a single command.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
