Evals in Action: From Frontier Research to Production Applications
- Source
- youtube.com
- Author
- OpenAI
- Date
Context
OpenAI built the GDPval because older academic benchmarks stopped showing whether models were improving at real work. In an OpenAI talk, its frontier-evaluations lead says that while o3 the team could not measure progress, because models were already near 100 percent on tests like AIME yet could not do real-world work.
GDPval has over a thousand tasks, drawn from the top US sectors and the knowledge-work occupations contributing most to wages, written by experts averaging 14 years of experience, such as making a property brochure or a 3D . Experts blind-compare a model's deliverable with a professional's and pick the one they prefer; the win rate is the share of times the model is preferred. The speaker reports GPT-4o scoring below 20 percent in spring 2024 and GPT-5 nearing 40 percent.
The speaker stresses limits: tasks are well specified with clear inputs and outputs, so results do not show jobs are automated; real work also involves deciding what to do, prioritising and responding to feedback. These figures are OpenAI's own.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- Training — The process that adjusts a model using examples and a measure of how well it performs.
- Model — A computational system that uses learned patterns to produce an output from information supplied to it.
Checking sign-in…
Loading comments…






