Link to blog:

LangChain measured how many of an agent's calls a router sends to its most expensive model, and on its own benchmark the share was small. It ran its Deep Agents evaluation suite, 145 multi-step tasks covering tool use, retrieval, file operations and long-context summarization, through NeMo Switchyard, an open source routing library from NVIDIA. The router sent 93% of model calls to Nemotron 3.5 Lightning, a 30-billion-parameter model, and 7% to Anthropic's Claude Opus 4.8. Compared with running Opus on everything, cost per run fell 74%, from $11.45 to $3.00, while accuracy dropped from 86.0% to 80.0%, according to the benchmark write-up. Even at 7% of calls, Opus still accounted for 68.4% of the routed bill.

LangChain tested Switchyard's escalation mode. Every task starts on the cheap model, and a small judge model (Gemini 3.1 Flash Lite in LangChain's configuration) reads each completed turn and votes on whether the agent is on track. After two consecutive negative votes, the task moves to Opus for the rest of its run and the judge stops checking it. The judge is a real cost: it runs on every turn until a task escalates and took 21.2% of routed spend. NVIDIA's introduction to Switchyard describes the general idea of sending each request to the model best suited to it; this benchmark measures one concrete setup of that idea on one workload.

The result shows routing is cheaper than a frontier model alone, not that it beats the cheap model. Nemotron alone scored 77.7% for $0.72 a run. Routing scored 2.3 points higher at 4.2 times the cost, and LangChain says that gap is smaller than the roughly 2.7 points its runs vary on their own, so it cannot claim routing outperformed the cheap model. The suite is also close to saturated, with only 8 points separating the small and frontier models, which leaves routing little room to prove itself. LangChain's case for a router is that in production you cannot tell in advance which requests are hard. If minimum cost is the goal and traffic resembles this suite, it recommends the cheap model alone.

LangChain also gives a quick test of whether routing can pay at all: divide the judge's cost per run by the price gap between the two models to get the share of turns that must go to the cheap model. For this pairing that was $0.64 over $10.73, or 5.9%, far below the 93% achieved. When two models are close in price, the required share can exceed 100%, and routing cannot save money unless the cheap model is self-hosted. The test only rules routing out on cost; whether a router makes good choices on a given workload still has to be measured.