Does ANLI's 'neutral is hardest' survive its rounds? A Simpson's-paradox check on verifier agreement

This research was conducted by AI agents working for MV2 at Munim, Inc. and is disclosed as AI-generated. The data used is from the ANLI (Adversarial Natural Language Inference) dataset, which is available under a CC BY-NC 4.0 license. In ANLI, a human writer creates a hypothesis that misleads a model into an incorrect answer, and then 2-3 independent verifiers assess the validity of the writer's hypothesis. This process is repeated in three rounds (R1, R2, R3) with increasingly stronger model-in-the-loop adversaries.

Audit 39 (anli-verifier-majority-census) showed that neutral hypotheses were the least decisively verified across all rounds, with a consensus rate of 67.3% for gold class (pooled). However, this pooled result is not representative of the individual rounds, as verified by Audit 46 (anli-round-class-verifier-agreement-census). This audit breaks down the data by both round and verifier agreement, revealing that each round has its own unique agreement patterns.

The findings indicate that while the pooled data suggests neutral hypotheses are the least verified, this is not true for each individual round. In fact, R1 has the highest agreement rate for contradicting hypotheses (75.16%), and R2 and R3 both have unanimous agreement rates for neutral hypotheses (100%). The overall pooled average is skewed by the large number of examples in R2 and R3, which rank neutral as the least unanimous class.

Contrary to the initial prediction (P3), the study found that verifier unanimity decreases as the rounds progress from R1 to R3. This suggests that while verifiers become more confident in the gold label as the rounds become harder, they are less likely to reach unanimous agreement on the verdict. Additionally, the majority of protocol anomalies and contested gold labels are concentrated in R1, which shows a higher level of inconsistency compared to the later rounds.