Why Frontier Models Fail as Buy-Side Analysts - Janie Gu, Google DeepMind & Suyeol Yun, LinqAlpha

Shows that many published benchmark scores are unverifiable or gameable, so treat leaderboard numbers with caution and consider building task-specific evals for your own domain.

• Janie Gu, who leads Kaggle Benchmarks at Google DeepMind, gives three reasons most AI benchmarks are broken: they don't reflect real work, they're expensive to keep current, and their scores can't be verified.
• MandateBench, built by LinqAlpha with Google DeepMind and Kaggle and presented by LinqAlpha head of AI Suyeol Yun, tests whether a model can go from a fund's mandate to a full investment memo as a buy-side analyst would.
• MandateBench grades a model's reasoning, not just its final answer, and the talk makes the case for companies building benchmarks for their own work.

• benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
• eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.

• videoDo Not Let Models Invent Numbers - Vinoo Ganesh, Kepler
• videoAI Writes the Code. So What's Left for Engineers? - Laurie Voss, Arize AI
• videoHow to Build Financial Agents That Work Across the Web - Jay Sahnan & Arik Bird, Browserbase
• videoYour Agent Fix Just Broke Something Else - Sofia Jakovcevic, Arize AI

• articleGames as Model Eval: 1-Click Deploy AI Town on Fly.iofly.io
• videoBenchmaxxing: The Gap Between Benchmark Scores and RealityAI Engineer
• videoInside the Race to Measure Frontier Intelligencea16z

Checking sign-in…

Loading comments…