Intel / podcast

Anthropic’s first technical PM on token maxing, the jagged edge, and living in the future | Dianne Penn

Source: lennysnewsletter.com
Author: Lenny Rachitsky
Date: Published Jul 26, 2026

Context

Vague complaints about an AI model become actionable when a team can reproduce the failure. In an interview, Anthropic product leader Dianne Penn recalls investigating reports that early Claude models failed to follow instructions. Many complaints concerned JSON that did not follow the requested format. JSON is a structured way for software to exchange data. The team assembled roughly 30 to 40 failing examples into an it could rerun on later models.

That turns a broad request to improve the product into a concrete behavior researchers can test. Penn still sees a role for product requirements documents in aligning teams and exploring ambiguous new products. Her account describes evaluations as a way to specify and measure a known user problem, alongside the broader work of deciding what to build.

Terms in this piece · Glossary

eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
MCP — The Model Context Protocol — an open standard that lets any AI assistant plug into any tool or data source without custom integration code.

Key quotes

“In 2023 when I started, nobody said anthropic and clod and coding in the same sentence.”
Dianne Penn

“You have to sweat the tokens as much as you sweat the pixels.”
Dianne Penn

“The evals, we actually have a saying on the team of evals are the new PRDs.”
Dianne Penn

“Because you can't improve what you can't measure.”
Dianne Penn

“no matter how far you go there's always another level”
Dianne Penn

Read the source www.lennysnewsletter.com