What Do You Do While AI Codes? I Make Mine Argue With Itself.

Watching the AI agent type has long been a common experience for developers. The act of waiting for the code to generate often leaves many of us idle, scrolling through the diff, nodding along to the code we don't fully understand, and promising to review it thoroughly later. Many have voiced this struggle in the dev.to community, and it's clear we're not alone in this experience.

To fill this wait time with something productive, the author created a system where a second model is designed to challenge the first model's work. The goal was to make the most of the time spent waiting for the AI to generate code. However, the author quickly realized that most of what they built was more of a spectacle than a useful tool.

When they ran a test by having two language models debate each other on a pull request, they found that the second model was not actually analyzing the code but simply replaying pre-generated text. The debate they witnessed was essentially a playback of a pre-written argument, not a genuine review process.

This realization highlights a significant flaw in many current AI second opinion systems. Most workflows involve Model A reviewing the artifact, and then Model B reviewing the artifact along with Model A's output. This creates a validation effect rather than a true independent review. Model B becomes anchored to Model A's verdict, instead of conducting its own analysis.

This mirrors the human tendency to be swayed by social pressure rather than rigorous evaluation. True independence is a crucial safeguard in human review processes, and it should be implemented in AI workflows as well.

The author's solution to this problem is called AdversarialDebate. In this system, Model B cannot see Model A's output until it has fully committed to its own independent position. The models analyze the code in parallel, commit their verdicts with structured claims and evidence, and then engage in a bounded debate where each objection must reference a specific counter-claim, and each claim must cite specific parts of the artifact.

This process ensures that the disagreement is preserved and documented rather than collapsed into a single verdict.

In a field test on 70 actual pull requests, 411 debates were run, and the results showed that 89% of the debate claims matched the actual issues found in the PRs. However, the author also fixed 13 bugs before feeling confident in the system, including issues related to CSV parsing errors and routing problems. These bugs were not architecture-level issues but rather implementation-level bugs that could have been caught earlier in the review process.

The author found that the pairing of models matters more than their individual capabilities. In their tests, GPT-4o-mini paired with Mistral Small 3.2 produced the best debates, with genuine concessions and resolutions. This suggests that diversity in training objectives plays a crucial role in the quality of debates. The author theorizes that models from different labs with distinct training objectives are more likely to produce meaningful disagreements, rather than rubber-stamping each other's conclusions.

While the author found that 88.7% of the debate claims matched the documented issues in the PRs, they acknowledge that false negatives are still a concern. If both models miss a real issue and converge on a "looks fine" verdict, it may seem like everything is fine until the issue surfaces downstream. This highlights the need for a higher bar for false negatives, especially in narrative domains where ground truth can be fuzzier.

Generic prompts that work well for code may not translate as well to more complex narratives, and requiring two independent models to both miss an issue is a higher standard than a single-pass review.

In conclusion, the author has shifted their approach while the AI codes. Instead of passively watching, they now let a second model challenge the first model's work while they focus on reading failure modes, writing evaluations, and deciding what constitutes good code. This active approach makes the wait time more productive and less lonely than watching a cursor. The author encourages others to explore this method and share their experiences with using AI to assist in the coding process.