Stanford CS329A Self-Improving AI Agents | Part 8 | Agentic Evaluations and Long Horizon Tasks
- Source
- youtube.com
- Author
- Stanford Online
- Date
Context
A Stanford lecture on evaluating agents explains the METR time-horizon measure, which describes progress in units of human work time. METR collects tasks from seconds-long actions to research tasks of up to eight hours and records how long skilled professionals take on them. For each model, the lecture says, METR fits a curve of success rate against human completion time, then reports the task length at which the model succeeds a chosen share of the time, such as 50 or 80 percent. The measure comes from the paper Measuring AI Ability to Complete Long Software Tasks, and the lecture walks through it.
The lecturer flags a limit: the baseline professionals have about five years of experience and may underestimate how hard a task is for a model, so human time is an imperfect stand-in for model difficulty. The lecture pairs it with OpenAI's GDPval, which compares model work with experts on economically valuable tasks, and says the two metrics show different trends. Other lectures in the course cover test-time compute and planning.
- Model — A computational system that uses learned patterns to produce an output from information supplied to it.
- test-time compute — Spending more computation when the model answers — thinking longer, trying multiple attempts — to buy accuracy without training a bigger model.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
videoStanford Webinar - Building Trustworthy AI: Navigating Security in the Agentic Era
videoStanford CS547 HCI Seminar | Autumn 2025 | HCI Systems in the Age of AI Code Generation
videoStanford CS25: Transformers United V6 I Advancing Science and Medicine with Collaborative AI Agents
videoStanford CS153 Frontier Systems | Andreas Blattmann from Black Forest Labs on Visual Intelligence
Checking sign-in…
Loading comments…


