Vibeleaderboard
← All Intel
Intel / video

Stanford CS329A Self-Improving AI Agents | Part 8 | Agentic Evaluations and Long Horizon Tasks

Source
youtube.com
Author
Stanford Online
Date

Context

A Stanford lecture on evaluating agents explains the METR time-horizon measure, which describes progress in units of human work time. METR collects tasks from seconds-long actions to research tasks of up to eight hours and records how long skilled professionals take on them. For each model, the lecture says, METR fits a curve of success rate against human completion time, then reports the task length at which the model succeeds a chosen share of the time, such as 50 or 80 percent. The measure comes from the paper Measuring AI Ability to Complete Long Software Tasks, and the lecture walks through it.

The lecturer flags a limit: the baseline professionals have about five years of experience and may underestimate how hard a task is for a model, so human time is an imperfect stand-in for model difficulty. The lecture pairs it with OpenAI's GDPval, which compares model work with experts on economically valuable tasks, and says the two metrics show different trends. Other lectures in the course cover test-time compute and planning.

Terms in this piece · Glossary
  • Model — A computational system that uses learned patterns to produce an output from information supplied to it.
  • test-time compute — Spending more computation when the model answers — thinking longer, trying multiple attempts — to buy accuracy without training a bigger model.
  • eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Read the source www.youtube.com
More from Stanford Online
Recommended reads
Comments

Checking sign-in…

Loading comments…