Offers a reproducible way to compare models across benchmarks that saturate, enabling long-run capability trend and forecasting analysis. Code is available to rerun it.
Key takeaways · AI-distilled
Epoch argues single benchmarks only separate models in their middle range, since very weak and extremely weak models both score 0% and strong models both hit 100%, which hides whether progress is speeding up or slowing.
The framework gives each model one capability number and each benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → one difficulty number, like Elo ratings for chess players and puzzles, plus a per-benchmark slope for how fast it saturates.
Scores map to capability through an S-curve with three regimes: near the random baseline when capability is far below difficulty, roughly proportional growth near parity, and saturation near 100% when capability far exceeds it.
Using data from Epoch's benchmarking hub, the authors stitch together around 40 benchmarks to estimate capabilities for around 200 models, letting models be compared even without shared benchmarks.
The authors acknowledge that reducing a model's capabilities to a single number is an oversimplification, while arguing the approach is still useful.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
