Asking how capable an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → is can conceal several different questions. This METRresearch note distinguishes the score achieved with a fixed budget from the cost of reaching a specified score. Those measures can favor different systems when giving an agent more time or computing resources continues to improve its result.
Comparisons with people add another requirement: the resources spent must be expressed in comparable units. Human working hours and model tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → do not become equivalent simply because both appear on a chart. The note organizes these choices rather than recommending one universal metric, making the question behind a capability claim easier to identify.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
