← All IntelClip / OtherBig vs small orchestrator and in/out-of-distribution routing risk
From The State of Model Routing — NVIDIA, Cognition, OpenRouter · ≈13:52
“Like if you run terminal bench on Opus and Haiku, like Opus will do about three times better at 1/10 the cost of Haiku, even though Haiku's significantly cheaper per token.”
“if it's out of distribution small models may actually increase your cost because of how often they'll like call tools and how crazy their loops will be.”
What’s in it
- Debates whether the orchestrating model should be big or small
- Explains why caching flips the cost math between big and small models
- Breaks down a terminal bench test showing Opus beats Haiku 3x cheaper
Clip transcript
router guy. >> [laughter] >> Yeah, um I think those are these are like important points and one of the biggest debates I think we have internally is whether that outer model that's doing the orchestration should be the big model or the small model. You get like very different results depending on your choice and in fact and it's not even clear what the pricing impact would be because if your outer model that's doing the orchestration is the big model it can leverage its caching to like make more of its decisions and it's caching is going to be like a dramatic price savings compared to the small models caching a lot of the time especially for like perform for you know issues that are on the bright line like zooming out a little bit I think the what you want from all the models out there when you do model fusion is to benefit from all the data that is being trained on across all the labs and and not just the data from one lab right or one source and a model is just like a combination of like the data and its understanding of the data both its compute and the quality of its RL so long term I think you want you want models where they know that oh this is like in distribution like this is in my data you can use small models pretty easily and get a cost savings but if it's out of distribution small models may actually increase your cost because of how often they'll like call tools and how crazy their loops will be. Like if you run terminal bench on Opus and Haiku, like Opus will do about three times better at 1/10 the cost of Haiku, even though Haiku's significantly cheaper per token. So it really becomes a huge problem if
Comments
Sign in to comment.
Loading comments…