Vibeleaderboard
← All Intel
Intel / article

AI R&D-relevant model capabilities are improving rapidly and are getting harder to measure

Source
metr.org
Date

Context

METR, which runs independent evaluations of what AI models can do on their own, says in its November capability update that the measurements it once used to argue frontier models were safe no longer settle the question. Its main measure has been the time horizon: the length of task, counted in the hours a skilled person would need, that a model completes with a 50% success rate. When METR introduced the measure in March 2025 the best model reached about an hour, and within a year models had saturated its Time Horizon 1.1 suite, which includes tasks up to 30 hours. On a new, harder suite called TH2, the current public models it tested, up to Anthropic's Opus 5.5 and OpenAI's GPT-6 Astra, reach an estimated time horizon of up to 27 hours on the automatically scored tasks, and they beat the experts METR recruited on the three tasks for which it tried to collect human baselines. Across both suites, time horizons are doubling roughly every 3.6 months.

The difficulty is that models look strongest on the kind of task that most resembles their training. METR worries that its cheap, automatically scored tests now have much in common with the reinforcement learning environments developers train models on, so a high score may overstate the broader ability the test is meant to stand for. To check, it estimated a Lifecycle Cost for each task, the cost of building it as a contained environment and then running and scoring it 100 times, using Opus 5.5 to produce the estimates. Tasks that cost more, for example because experts must score them, they need GPUs or they run for 48 hours, were harder for models than their length alone predicts, and on the costlier group of tasks model time horizons were 2.9 times shorter. Models also showed weaker judgment where feedback is slow: none of the evaluation tasks that agents built for METR was useful enough to include, and model judges scored work by equally strong or stronger models too generously, with Opus 5.5 rating GPT-6 Astra's tasks about 0.2 higher on a scale of 0 to 1 than METR's human scorer did.

Those weaknesses buy less time than they might seem to. On METR's data, models improved at a similar rate on costly and cheap tasks, and each doubling of lifecycle cost delayed the trend by about 4.6 months, although METR cannot say how many doublings more expensive the work that matters would be. Because public models can already hack services and get past restrictions, METR says it can no longer argue that risk is very low because models fail at verifiable software tasks, and it concludes that public evidence cannot rule out drastic AI R&D acceleration in the next couple of years, which it defines as progress too fast for humans to understand and react to catastrophic risks. The report is open about how uncertain this is: human baselines directly support only 9% of its estimates of how long tasks take people, its composite index shows no clear acceleration across the public frontier, and METR says it does not know which capabilities such acceleration would require.

Key takeaways · AI-distilled
  • METR finds 50% time horizons on its combined TH1.1 and TH2 suites doubling about every 3.6 months (95% CI 2.1 to 4.9), close to the 3.8 months OpenAI's data implies for internal research tasks. Naive extrapolation hits its longest 200-hour threshold in July 2027.
  • METR's new Lifecycle Cost metric estimates what a task costs to build and run 100 times as an RL environment. At a fixed task length, each doubling of that cost made tasks harder for models by very roughly as much as halving the 's time horizon.
  • On 15 past Al Zimmermann optimization contests, Fable 5.1, Opus 5.5 and GPT-6 Astra placed at the 96th to 98th human percentile with 48 hours each, yet about four humans still beat them in the median contest.
  • Model judges were overgenerous on tasks built by equal or stronger models: Opus 5.5, the judge closest to METR's human scorer, rated GPT-6 Astra's tasks about 0.2 points (on a 0 to 1 scale) higher, and no agent-built eval was useful enough for METR to include.
  • Agent swarms beat best-of-k controls mainly on tasks with clear, fast feedback such as NanoGPT speedrunning. On hidden-test tasks like Train a Program, larger swarms overestimated their own scores more, and the swarm runs cost METR about $500k in .
Terms in this piece · Glossary
  • sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
  • inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Recommended reads
Comments

Checking sign-in…

Loading comments…