When strong models agree on a different GSM8K answer, the answer key is usually wrong

A recent study analyzed whether the consensus of multiple strong AI models could detect errors in the GSM8K answer key. When six or more of the ten most accurate models agreed on a number that did not match the key, the researchers considered it a potential issue. Out of the 22 flagged questions, 19 were correctly identified by GSM8K-Platinum as needing correction or removal, which is 86.4% of the flagged questions.

The research found that the flagging system worked better for identifying wrong answers than ambiguous or flawed questions. For wrong answers, the flagging system successfully identified 5 out of 6 issues, and the number the models agreed on matched the corrected answer provided by GSM8K-Platinum. However, for flawed or ambiguous questions, the flagging system only identified 14 out of 84 issues, indicating that it is less effective for these types of problems.

The study also found that even though 10 out of 10 top models may sometimes reach a wrong conclusion, individual model agreement can still be a useful tool for identifying errors. However, it is essential to double-check the flagged questions to ensure accuracy. The researchers recommend using the strongest available models for this check, as the weaker panels had lower success rates.

For ambiguous questions, they suggest using a broader filter, such as any strong model getting the answer wrong, and then having a human review the question.