Transcript
Run terminal bench on Opus and on Haiku and Opus scores about three times better at a tenth of the cost, even though Haiku is far cheaper per token. Alex Atallah's point is that a small model pushed outside its training distribution thrashes, calling tools in loops until it costs more than the expensive model ever would. That inverts the obvious version of model routing, where you send each task to whichever model benchmarks best on it. Walden Yan calls that approach fragile for exactly the reason agents make it worse: a session starts as a question about a codebase, becomes a feature request, then becomes live debugging, and the model you picked at the start is stranded. Cognition's answer keeps a frontier model planning and delegates the implementation, which cut the cost of Fable level intelligence by 40% while going deeper, because a cheaper model can afford to spin off three sub agents to explore a codebase. They also avoid sub agents in favor of one sidekick with a continuous running context, so the KV cache stays warm and cached tokens cost roughly ten times less. Compaction, Yan argues, is worth doing for intelligence rather than cost, since compacting forces a cache miss and model quality falls off a cliff well before the advertised million token window. The most telling story is OpenRouter's: its auto router sat almost unused for two years until openclaw began sending heartbeats every ten minutes, creating one popular app with two completely different intelligence needs. Speaker info: Nader Khalil, moderator (NVIDIA): - https://x.com/naderlikeladder - https://nader.coffee Walden Yan (Cognition): - https://x.com/walden_yan - https://www.linkedin.com/in/waldenyan Alex Atallah (OpenRouter): - https://x.com/alexatallah - https://openrouter.ai Tanay Varshney (NVIDIA): - https://www.linkedin.com/in/tanayvarshney Carter Abdallah (NVIDIA): - https://x.com/Baxate - https://www.linkedin.com/in/carter-abdallah Timestamps: 0:00 - Welcome and the multimodel premise 1:16 - Panel introductions 3:24 - How Devin Fusion beats the frontier models 4:25 - Let the frontier model plan and delegate the work 6:31 - Jagged capabilities: no one model wins everything 9:42 - Why naive task based routing is fragile 11:48 - Sharing context without paying for it twice 13:56 - Should the orchestrator be the big or the small model 16:01 - In distribution versus out of distribution 19:12 - Training models to collaborate 20:12 - Flex Run and flexible model sizes 22:24 - Lossy context and the systems you fall back to 26:41 - How a heartbeat created the auto router 29:43 - Routing between local and cloud 31:51 - Compaction versus routing 32:55 - How a small model signals it is out of its depth 35:00 - Cache duration and self hosting economics 40:12 - Are prompts portable across models 43:20 - Is the router a product or plumbing