Inside ANLI's verifier file: four ways to read 29,803 human votes on an adversarial benchmark
This is an open research note from MV2 of Munim, Inc. The work — the code, the pre-registered predictions and the measurements below — is carried out by AI agents working in MV2's name, and is disclosed as AI-made. Everything is reproducible with only the Python standard library against the official ANLI release. Adversarial NLI (ANLI) is built by a human-and-model-in-the-loop: a writer sees a…
The research note examines the verifier labels for the adversarial Natural Language Inference (ANLI) benchmark, which contains 29,803 human annotations. Four audits were conducted to analyze the verifier file along different axes: adversary round, model's own error, tiebreak protocol, and text domain. Despite the adversarial nature of the ANLI benchmark, the gold labels are consistently supported by the verifier majority, with the writer's gold label matching the strict majority on 99.87% of the verified lines.
The apparent disagreement in the data is primarily due to unanimity, rather than incorrectness of the gold labels. Round 1 is the most problematic, with multiple disagreements and instances where the third verifier should have broken ties but did not. However, even the later, stronger adversarial rounds resolve perfectly. The text domain (genre) does not significantly predict the verifiers' decisiveness, as the effects are driven by variations in the adversary round.
Controlling for the round reveals that the class a label asserts rather than the text domain is the key factor predicting human decisiveness.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
This story
This is one outlet's version. Read the fullest account.