Public benchmarks leak into training data, and most are uncalibrated: nobody knows whether going 84% to 85% is as hard as 40% to 41%. Many also contain wrong answer keys, so the achievable ceiling sits below 100%.
MMLU-Pro asks things like the mean cranial capacity of Homo erectus. Any single question tells you almost nothing, yet the benchmarks move up together, which suggests they are all reading some shared underlying ability factor.
GDPval's method is the template worth copying: experts with ~14 years in a field write tasks that take a human 4-7 hours, both models and paid human experts complete them, and a third blinded expert group grades, at over an hour per task.
The results come out jagged by occupation. Top models beat human experts at software development and personal financial advice, but lost clearly to pharmacists, industrial engineers and real estate agents.
To probe a model's disposition, pitch it a deliberately weak business, then rate it ten times because answers vary. Mollick's drone-delivered guacamole drew consistent scores within each model but wildly different ones across models.
Why it matters
If you're picking an AI model for real work, this shows why leaderboard scores mislead and gives you concrete alternatives—vibes-testing, GDPval-style real-task benchmarks, and ambiguous-judgment probes—to assess models against your actual needs.
Key quotes
“Every benchmark has flaws, but they are all trending the same way - up and to the right.”
“You need to know specifically what YOUR AI is good at, not what AIs are good at on average.”
“You need to systematically test your AI on the actual work it will do and the actual judgments it will make.”
“So good benchmarks help you figure out the shape of what we called the Jagged Frontier of AI ability, and also track how it is changing over time.”
