Vibeleaderboard
← All Intel
Intel / video

Evals in Action: From Frontier Research to Production Applications

Source
youtube.com
Author
OpenAI
Date

Context

OpenAI built the GDPval because older academic benchmarks stopped showing whether models were improving at real work. In an OpenAI talk, its frontier-evaluations lead says that while o3 the team could not measure progress, because models were already near 100 percent on tests like AIME yet could not do real-world work.

GDPval has over a thousand tasks, drawn from the top US sectors and the knowledge-work occupations contributing most to wages, written by experts averaging 14 years of experience, such as making a property brochure or a 3D . Experts blind-compare a model's deliverable with a professional's and pick the one they prefer; the win rate is the share of times the model is preferred. The speaker reports GPT-4o scoring below 20 percent in spring 2024 and GPT-5 nearing 40 percent.

The speaker stresses limits: tasks are well specified with clear inputs and outputs, so results do not show jobs are automated; real work also involves deciding what to do, prioritising and responding to feedback. These figures are OpenAI's own.

Terms in this piece · Glossary
  • eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
  • Training — The process that adjusts a model using examples and a measure of how well it performs.
  • Model — A computational system that uses learned patterns to produce an output from information supplied to it.
Read the source www.youtube.com
More from OpenAI
Recommended reads
Comments

Checking sign-in…

Loading comments…