Notion replays sampled real tasks across models in isolated sandboxes from identical workspace state, then judges outputs by pairwise comparison with calibration. It is a reusable pattern for evaluating agents and model routers on messy work.
Key takeaways · AI-distilled
Notion built its Auto evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → set by taking the 20 largest clusters of real Notion AI tasks, such as CRM upkeep and support replies, and randomly sampling hundreds of tasks from each, using data only from customers who opted in to data sharing.
Notion found single LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → judges unreliable even with detailed rubrics, noting judges could favor their own provider, so it had an ensemble of frontier models pick pairwise winners by majority and converted those preferences into Elo scores.
To scale beyond costly pairwise runs, Notion tuned one LLM judge's model, prompt and thinking level until its pointwise scores matched the Elo rankings; Notion reports the two methods agreed more than 80% of the time.
Notion reports its calibrationHow well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.Full definition → judge correlated positively with customer thumbs-up and thumbs-down feedback, usually ranking liked responses higher and flagging disliked ones as low quality.
Notion estimates Auto uses 5, 10 and 25 credits for simple, medium and complex tasks versus 50, 85 and 150 for Opus 5.5, and says Auto reaches roughly 90% of frontier-model performance at about 90% lower cost.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
