Why Tejal Patwardhan stopped underestimating the models - Episode 21
- Source
- youtube.com
- Author
- OpenAI
- Date
Context
Tejal Patwardhan, who works on frontier evaluations at OpenAI, says on the company's podcast that a loses value once models score near 100%, because it can no longer tell models apart, like using a high school math exam to compare two geniuses. Patwardhan's team responded by building tests closer to real work.
One is GDPval, which asks models to do tasks drawn from Bureau of Labor Statistics occupations, such as a financial analyst's diligence work or a legal memo. Patwardhan says an early model scored under 20% against human work, and that OpenAI's models are now the best on it. Patwardhan also says GDPval is probably too easy now, since each prompt spells out the task in hundreds of words, and the next step is tasks with the ambiguity of a manager's request. These are Patwardhan's account; the conversation gives no current scores.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
Checking sign-in…
Loading comments…






