Intel / article

Route AI Agent Workloads Across Models with NVIDIA NeMo Switchyard

Source: Michelle Horton
Author: Michelle Horton
Date: Published Aug 11, 2026

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
model routing — Sending each request to a model chosen by the difficulty of the task, rather than using one model for everything.

Why it matters

inside loops is becoming a standard cost lever, and this is an open implementation from the vendor rather than a blog post about the idea.

Key quotes

“For example, an agentic task may need classification for one step, reasoning for the next, and a smaller model for routine follow-up tasks. Sending every request to the largest model can increase cost and latency, while sending every request to a smaller model can reduce quality on complex tasks.”
Michelle Horton

“While DeepSeek V4 has the highest overall accuracy in this example, it isn’t the best model for every task group. For instance, Kimi K2.6 is better suited to the ML and RL task groups, while Qwen3.5 397B A17B is preferable for math and science.”
Michelle Horton

“For each turn, the stage router examines recent tool activity to decide how much model capability the agent needs. Severe errors, repeated unproductive work, or prolonged exploration push the turn toward the capable model. Steady writes and edits, especially once tests are passed, favor the efficient model.”
Michelle Horton

“Across five runs, routing requests between NVIDIA Nemotron 3.5 Lightning and Claude Opus 4.8 with the escalation router delivered a 74% cost reduction compared with a frontier-only baseline across five runs, sending just 7% of calls to the frontier model, at a measured ~6-point accuracy tradeoff.”
Michelle Horton

“On FrontierCode Main, Cognition’s benchmark for production-grade coding tasks, the implementation routed between Opus 5 and Kimi K2.7. It delivered near-frontier performance, achieving 50.6% at a $3.11 mean cost—within 2.8 percentage points of Opus 5 accuracy at approximately 28% lower mean cost.”
Michelle Horton

Read the source developer.nvidia.com