Impact of modeling assumptions on time horizon results
Source
metr.org
Date
Context
An AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →'s time horizon is a statistical estimate, not a stopwatch reading of how long it worked. METR's methodology note examines how task lengths and observed success rates are fitted into that estimate. As stronger models approach the limits of the available task set, fewer difficult examples constrain the result.
Reasonable changes to the fitted curve can lower the estimated horizon at 50% reliability while raising it at 80% reliability. That is a change in the analysis, not necessarily a change in the agent. The note identifies the mix of tasks as a particularly large source of uncertainty, so the confidence interval and success threshold are essential parts of interpreting a headline duration.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
