I Added a Fourth Model Mid-Run. It Changed What My Field Test Could Prove.
Latest release: v0.2.2 — Aug 29, 2026 I did something I usually try hard not to do in a field test. I changed the design after it had already started. Halfway through validating AdversarialDebate, I realized the model set was too narrow to answer the most important question in the project. So I added a fourth model in the middle of the run. That was messy. It wasted work. It made the corpus…
The author admits to a mistake in their field test that they usually avoid. During the validation of AdversarialDebate, they discovered the model set was too limited to answer the project's most crucial question. To address this limitation, they introduced a fourth model, Mistral Small 3.2, into the middle of the evaluation run. This alteration was initially messy, causing waste and inconsistency in the corpus. However, it turned out to be one of the best decisions in the entire release.
The setup initially included three models: GPT-4o-mini, Gemini 2.5 Flash, and DeepSeek-V3. This configuration produced three useful pairings, with the diverse pair (GPT + Gemini) outperforming the others. The author realized the experiment could only observe part of the diversity spectrum using these three models, which limited its ability to determine the impact of maximum diversity.
The author added Mistral Small 3.2 to the test, creating three new pairings: GPT + Mistral, Gemini + Mistral, and DeepSeek + Mistral. The most significant pairing, DeepSeek + Mistral, produced the strongest diversity pairing in the run, covering the China + EU spectrum. This enabled the experiment to observe the full range of diversity, from homogeneous to strong diversity, which significantly improved the field test's quality.
The addition of Mistral revealed several key insights:
1. The strongest pairing, DeepSeek + Mistral, outperformed expectations, with a 0.982 average score and a 97% verdict rate.
2. Maximum diversity exhibited a failure mode, with 44 capitulation cascades and a 65% capitulation rate within the pair.
3. The homogeneous control, GPT + GPT, became more meaningful in the broader diversity context, indicating that weak diversity might be a specific failure mode rather than just a weaker result.
In summary, the fourth model proved crucial in revealing the limitations of the experiment and providing a more accurate and nuanced understanding of pairing behavior. This experience emphasizes the importance of field tests not only in generating data but also in uncovering whether an experiment can effectively answer the question it aims to address.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.