Three-Model Jury: The Price of Consensus
A single model's verdict is an opinion. Three models agreeing looks like evidence. In agent telemetry, that appearance is a signal trapped inside a design decision. The three-model jury is popular because it promises cross-validation. But before adopting it, you need to know exactly what consensus buys—and what it costs. The mechanics of a jury I treat each model as a witness, not a judge. The…
In the world of agent telemetry, the appearance of consensus from multiple models is often viewed as evidence of correctness. However, adopting a three-model jury requires understanding exactly what consensus truly provides—and what it costs. Each model functions as a witness, not a judge, receiving identical observations and constraints before delivering a structured recommendation.
Controllers then collect these witnesses and apply a policy, typically requiring at least two votes for safety-critical actions, with a fallback to human review.
Observability is crucial in this process. Each vote should be wrapped in a trace span, including the model's ID, name, action, rationale hash, and final decision. This allows for tracking not just which action won, but which witness dissented and why. While the upfront cost of a serial jury (where one model proposes, another challenges, and a third decides) is significant, involving at least two inference round trips, parallel juries burn roughly three times the tokens per decision, with retries further inflating these costs.
More insidiously, correlated blind spots can arise when three models, trained on overlapping public data, share the same biases. If a prompt contains urgency cues, all three may vote for the unsafe path, merely because they read from the same corrupted script. Thus, agreement does not equate to correctness; it may simply be three witnesses echoing the same flawed reasoning.
Another concern is quorum gaming, wherein model outputs are non-deterministic. Rerunning the same juror can flip a vote, leading to decisions based on luck rather than reliability. Abstention collapse also poses a risk; a model may refuse to act due to ambiguous constraints, causing quorum failure and routing the case to an already overloaded human. While this is a safe failure, it introduces an additional cost that must be tracked distinctly from disagreements.
The detective's protocol emphasizes trusting the minority report over the majority. The losing model's rationale often uncovers ambiguities the majority overlooked. Logging disagreement rates by state type, rather than by individual model, reveals where consensus is meaningful and where it is merely cheap theater. Instrumenting outcomes to compare the jury's final decision against later success or failure signals helps determine if the jury is adding genuine safety or merely latency and token usage.
Ultimately, a three-model jury serves primarily as an observability device before it can be genuinely effective as a safety measure. The true cost extends beyond inference expense, encompassing false confidence. If logs fail to capture dissent, abstentions, and correlated blind spots, the system is not functioning as a jury but as three independent guesses.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
