Picking the right model
- Source
- youtube.com
- Author
- Claude
- Date
Context
A small test built from your own tasks matters more for choosing an AI model than public benchmarks. That is the advice of Lucas, a member of Anthropic's applied AI team, in a talk on picking the right model. Lucas says benchmarks like give direction but rarely match a real workload, for instance by omitting the languages a team uses.
Lucas frames the choice around quality, and cost, and argues the right model has the lowest cost per successful outcome, not per token. An evaluation is a set of tasks with inputs and success criteria; for agents, grade the steps as well as the final answer, using a model as judge for flexible matches (such as equivalent ) and code checks for fixed rules.
Listed pitfalls: run each task several times so noise isn't mistaken for signal, separate infrastructure failures from model failures by reading transcripts, and keep the test set representative using production traces. Prompts may need retuning per model; one tool under-triggered on Opus 4.5 and over-triggered on 4.6 with the same . This is a vendor employee's practical guidance, not tested evidence.
- SWE-bench — The standard benchmark for AI coding agents: real GitHub issues from real repositories, scored by whether the agent's patch passes the project's own tests.
- Latency — The time between asking for a result and having it, including the model's generation time, tool calls, retries and any queueing.
- Database and SQL — Persistent structured storage queried and changed through explicit operations.
- Prompt — Information or instructions supplied to a model; in a chat, the request you type is often called your prompt.
Checking sign-in…
Loading comments…






