Forecasting the Impacts of AI R&D Acceleration: Results of a Pilot Study
Source
metr.org
Date
Why it matters
Shows how experts translate hypothetical AI R&D evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → results into expected acceleration, exposing where forecasters disagree. Helps practitioners read capability benchmarks against economic consequences, though the sample is only 18.
Key takeaways · AI-distilled
Experts gave a three-fold acceleration of effective compute scale-up before 2029 a median 20% chance and superforecasters 8%. Both expected post-training gains, not physical compute, to accelerate fastest.
If AI beat human researchers on open-ended month-long research projects, the groups moved to medians of 20.5% and 18%. Superhuman RE-Bench scores were seen as uninformative, since many expect that benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → to saturate by 2027.
On extreme events such as global energy use halving or doubling in a year, superforecasters gave medians below 0.5% and experts over 18%. Conditioning on 3x or even 10x acceleration did not close that gap.
The study's 'effective compute' combines spending and algorithms: a model trained with twice the compute using algorithms that halve compute needs counts as a 4x scale-up.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
