Vibeleaderboard
← All Intel
Intel / video

Why Tejal Patwardhan stopped underestimating the models - Episode 21

Source
youtube.com
Author
OpenAI
Date

Context

Tejal Patwardhan, who works on frontier evaluations at OpenAI, says on the company's podcast that a loses value once models score near 100%, because it can no longer tell models apart, like using a high school math exam to compare two geniuses. Patwardhan's team responded by building tests closer to real work.

One is GDPval, which asks models to do tasks drawn from Bureau of Labor Statistics occupations, such as a financial analyst's diligence work or a legal memo. Patwardhan says an early model scored under 20% against human work, and that OpenAI's models are now the best on it. Patwardhan also says GDPval is probably too easy now, since each prompt spells out the task in hundreds of words, and the next step is tasks with the ambiguity of a manager's request. These are Patwardhan's account; the conversation gives no current scores.

Terms in this piece · Glossary
  • benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
  • eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Read the source www.youtube.com
More from OpenAI
Recommended reads
Comments

Checking sign-in…

Loading comments…