What We Learned by Reproducing 2,200 papers from ICML

Back in July, a hackathon brought together over 1,200 participants to attempt reproducing papers from the ICML 2026 conference. In 19 days, 6,816 logbooks were published, reproducing 2,226 papers, which was about a third of the conference. This hackathon was the largest attempted reproduction of a scientific conference, with 2,226 papers examined.

51% of the examined papers had at least one claim independently verified, with 266 papers fully reproduced and 632 partially reproduced. 3,978 individual claims were confirmed with real experiments. However, 23% of the examined papers had at least one falsified or contested claim, including 49 papers where all claims were falsified and 242 papers where independent reproduction teams reached opposite verdicts on the same claims.

Some papers stood out as well-reproduced, while others faced challenges such as toy-scale evidence, missing artifacts, or falsification claims. The hackathon highlighted the importance of careful verification and the potential for agents to aid in reproducing research at scale. Some notable findings include:

- The paging paper's claim of robustness Hk+O(1) being proven incorrect, with the true robustness being Hk+Θ(logk).

- A counterexample to the claim that attention's forward pass collapses to the origin, found by three independent teams at different steps.

- A discovery that ~66% of evaluated label positions in a transformer paper were EOS padding tokens, deflating the quality cost by threefold.