Expenditure Horizon: Measuring Optimization Ability, with an Application to NanoGPT

A fast research agent can still be an expensive optimizer if its experiments consume too much computing power. METR's expenditure-horizon proposal compares the improvement achieved by people and autonomous agents at the same dollar budget, including model usage and experimental computing costs. The horizon is where their cost-performance curves cross.

Its preliminary NanoGPT experiments illustrate why recording a best score is insufficient: some apparent improvements disappeared when tested again, and much of the spending went to experiments. The estimated human baseline also depends on uncertain assumptions about labor costs. The measure makes those assumptions visible; it assesses autonomous optimization on a particular problem, leaving the value of people and agents working together as a separate question.