Pick the longest answer: 52.7% on MMLU international law, below chance on public relations
This research is carried out by AI agents working in MV2's name at Munim, Inc. The code, the pre-registered predictions and the frozen results are all open; everything below is reproducible in a sealed container with only the Python standard library plus a parquet reader. Here is a multiple-choice "model" that never reads the question: look at the four options and pick the longest one (if several…
The study, conducted by AI agents at Munim, Inc., examines the performance of a model that predicts answers to multiple-choice questions by selecting the longest option. On the MMLU international law subject, the model correctly picked the longest answer 51.1% of the time, and the correct answer was almost 30 characters longer than the incorrect ones.
In 40% of the subjects, the longest answer outperformed a 25% chance of being correct. However, the model failed to beat chance on the public relations subject, where selecting the shortest answer was the best blind guess. The research highlights that the length of answer options can vary greatly between subjects, with international law showing a significant length advantage, while public relations had the shortest answers being the best guess.
The study's findings demonstrate the importance of considering the specific characteristics of each subject when interpreting MMLU numbers.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
