What it actually takes to build frontier models | Tulsee Doshi (Head of Product for Gemini)
Source
lennysnewsletter.com
Author
Lenny Rachitsky
Date
Why it matters
Describes how Gemini's product lead plans features ahead of model capability (a 60-20-20 rule) and evaluates models by hands-on testing, which shows how a frontier lab shapes what ships.
Key takeaways · AI-distilled
Tulsee Doshi describes a 60-20-20 rule for building on models: ship a product where about 60% works today, 20% works well enough with a visible path to improve, and 20% targets capabilities models only show glimmers of. She says the exact numbers are fungible.
For a first evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition →, Doshi suggests starting with as few as five prompts and growing to about 20 that stress-test a set of capabilities, with yourself as the rater. A small eval that separates model A from model B already gives useful signal before you scale it up, she says.
Doshi says end-to-end evaluation has to cover more than the model: the judge model's quality, production mitigations and filters, and how the model is served all change what users experience. She cites audio, where serving can hurt voice naturalness even when the model is good.
According to Doshi, Gemini's model behavior work starts from user feedback and a spec of what good looks like, which the team turns into rubrics for evaluation and then into data used for reward modeling.
Doshi recommends regularly retesting a agent harnessThe scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.Full definition → against new models, since forcing a newer model into structure built around an older one's limits can hamper what it delivers. She defines a harness as the system instructions, tools, skills and context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → the model can use.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
podcastInside the black box of AI: How AI models actually think | Emmanuel Ameisen (Anthropic)
podcastOpenAI’s Head of ChatGPT: We’re entering a new era of AI (again) | Tibo Sottiaux
podcastHow we built Grok Bot in a month | Roman Ugarte (SpaceXAI)
podcastAI’s third era: the rise of persistent AI coworkers | Tara Seshan (OpenAI’s product lead)
