New ‘First Proof’ exam reveals how AI-powered math is a pay-to-play sport
The benchmark confirms that some AIs are becoming exceptional tools for research-level math. But open-source models still struggle
The First Proof exam has revealed a significant disparity in the mathematical abilities of different AI models, highlighting the unequal competition in the AI-powered math landscape. The benchmark, created by a team of top mathematicians, aims to objectively assess the performance of publicly available AI models in research-level mathematics.
While some large language models (LLMs) are becoming increasingly potent research tools, proprietary subscription-based models from so-called frontier labs outperform open-weight models. Mohammed Abouzaid, a mathematician from Stanford University and a member of the First Proof team, notes that the gap between these two types of models is much larger in research mathematics than in other domains.
OpenAI's "slop drop" and the Navier-Stokes problem solution have accelerated the AI race to conquer math, but these achievements are far from controlled experiments. The First Proof team solicited problems from experts and selected 10, asking each model to answer them without human intervention. Three graders then evaluated the answers in a mock peer review.
The best-performing model, ProofCouncil, solved nine out of the 10 research problems with minor revisions required for publication in a peer-reviewed journal. However, the leading open-weight model, Kimi K3, failed to solve any of the problems. This stark contrast suggests that open models are desperately behind, and the First Proof team urges the community to invest in improving them.
Written by urgent.news from Scientific American's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
