← All IntelClip / OtherOpenRouter's cache-hit passthrough and dynamic model-switching tradeoff
From The State of Model Routing — NVIDIA, Cognition, OpenRouter · ≈25:07
“So open router is a like a marketplace for language models.”
“we try to like find the best model or best combination of models for the prompt and then when we see a cash hit, we will will like use up the duration of the cash and send the downstream customer like the the full savings of the cash hit.”
What’s in it
- Explains how OpenRouter routes prompts across many LLM providers
- Reveals OpenRouter passes cache-hit savings straight through to customers
- Teases an unreleased feature to swap models mid-cache for efficiency
Clip transcript
>> So open router is a like a marketplace for language models. We we exist at like we can't see into the KV caches of models unless we're running them ourselves, which is pretty rare. Um we we we do spend a lot of time optimizing for cash hits and we we like pass through cash hits directly to users. Um but like in terms of KV cash optimizations we we we can't do any of like specific work there. What we do for um model routing is we we try to like find the best model or best combination of models for the prompt and then when we see a cash hit, we will will like use up the duration of the cash and send the downstream customer like the the full savings of the cash hit. Um there's more work that we can do here where we could say, okay, this looks like something where there's significant benefit to switching the model right now, but you haven't used up the full cash. You still have like 2 minutes left. Um and it we think it's probably worth switching the model and losing the rest of your cash um and letting people kind of like tweak their tolerance for that behavior. Um we've we've been doing a little bit of that, but we haven't like exposed it to customers yet.
Comments
Sign in to comment.
Loading comments…