Notion's Token Town — Sarah Sachs, Notion
- Source
- youtube.com
- Author
- AI Engineer
- Date
Context
An unchanged price per model can still produce a higher bill if an uses more tokens to finish the job. In this talk, Notion's AI engineering lead argues for comparing complete task runs: their cost, response time, tool errors and result quality.
That changes model selection from a price-list comparison into a product-specific test. The speaker's proposed approach keeps multiple providers usable, evaluates less expensive models on the actual work, and executes predictable operations as ordinary code. The aim is to reserve costly model reasoning for steps that need it. This requires engineering investment, including handling the loss of cached when switching models.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- reasoning model — A model trained to think — generating extended internal reasoning before answering — trading time and tokens for accuracy on hard problems.
“your supplier is your competitor. I know very few people who have convinced me that that's not true.”
Sarah Sachs
“Stop trying to win that game and think about the best product that uses many models.”
Sarah Sachs
“When you triage an email inbox, if we're charging you to do that on Opus, we're ripping you off and ourselves.”
Sarah Sachs
“optionality is your leverage if you don't have the capability to walk at any point you are stuck”
Sarah Sachs
“you don't need an LLM to turn a CSV into a PDF”
Sarah Sachs
- 0:00Welcome to Token Town: AI poor vs AI rich
- 1:25Negotiating AI contracts as Notion's AI Anna Wintour
- 2:30The AI transformation journey and the system of record
- 4:46Why cost is the structural barrier
- 5:34The monthly model pricing traps
- 7:28Your supplier is your competitor
- 8:44Win on product, not token economics
- 9:50Not all traffic belongs on the frontier model
- 12:10Optionality is your leverage
- 13:25The auto model and the model agnostic playbook
- 15:04Open weight models for the middle
- 17:00CPUs over GPUs and governance
- 17:53Security: the lethal trifecta
- 19:46Live demo: orchestrating Notion agents into a PR
videoBetter AI Models Won't Fix Your Engineering Team — Nicholas Arcolano, Jellyfish
videoWhy AI Writing Still Reads Like Slop — Louis-François Bouchard, Towards AI
videoHow to Test AI Agents When No Benchmark Exists — Nathanael Weill, Air Canada
video1,368 AI Failures and What They Teach About Evals — Vishal Srivastava & Tanmay Sah, New York Life
Checking sign-in…
Loading comments…