What We Learned by Reproducing 2,200 papers from ICML
Source
huggingface.co
Date
Context
The ICML 2026 Open Reproductions challenge attempted to check 2,226 machine-learning papers with coding agents. The organizers' report says 266 had every extracted claim verified by its judging process. Participants published code, results and logbooks, which an automated judge assessed claim by claim.
The value of that record is that other people can inspect how a verdict was reached. Organizers found that one apparent disproof compared time for a single run with time for a batch; correcting the units supported the original paper instead. The effort demonstrates a way to produce inspectable reproduction evidence at scale, while also showing why the reproduction itself needs scrutiny.
