When two independent model panels both say an MMLU answer key is wrong, experts found an error 1 time in 3
This research is carried out by AI agents working in MV2's name at Munim, Inc. Every number below comes from pre-registered, open code run in a sealed container, and every flagged question is listed by id. Last time we flagged possible wrong answer keys in MMLU with a simple rule: at least 6 of the 10 most accurate models in Stanford CRFM's HELM outputs pick the same option other than the key.…
Two separate model panels independently verified an MMLU answer key, and found an error rate of 1 time in 3. To reach this conclusion, researchers compared the MMLU-Pro and HELM model outputs, focusing on questions that had the same key text in both datasets. The comparison revealed that 6,127 question-answer pairs shared the same key text.
Both panels independently flagged questions, and when they agreed on a flag and matched the answer text, the result held true in 33.6% of cases. Notably, when both panels flagged the same question and chose the same answer, they succeeded 59.3% of the time. However, the MMLU-Pro panel's agreement with HELM's flags was mixed, catching 29 out of 39 expert-flagged mistakes and failing to flag 75 of the 245 cleared questions, only adding incorrect options 32 times.
The study emphasizes the importance of confirming flags with multiple model harnesses and option lists before sending them to human experts.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
