← All IntelClip / OtherCompaction vs. routing: why compaction is about intelligence, not cost
From The State of Model Routing — NVIDIA, Cognition, OpenRouter · ≈31:26
“I would like never recommend using like these models past like 200K tokens, under 100K if you can.”
“Sorry, Anthropic if you're if you're watching.”
“The main reason we compact is actually intelligence.”
What’s in it
- Explains why compaction beats model-routing for managing long-context costs
- Reveals context windows advertised at 1M tokens are basically unusable past 200K
- Shows how compaction avoidably triggers costly cache misses on cheap tokens
Clip transcript
higher cost at a higher context length because your throughput slows down as the context gets deeper. So, rather than switching to a cheaper model, even if you have self-hosted models in data center, you can use compaction uh to bring your throughput back up. Uh have you guys uh thought about this part of it? Like compaction versus uh just routing because one is you have fewer tokens to work with. One is we have cheaper tokens. >> Yeah, I I think in practice um compacting alone doesn't solve the the cost or or throughput problems because a lot of times it's just like the uh the differential in like model intelligence and cost is just so big where also by the way, when you compact, you're taking a cache miss. So, you're actually then now like paying 10 times as much for the for those input tokens if you didn't compact. Um the main reason we compact is actually intelligence. Um all all these like multipliers they advertise some like insane context window, like a million tokens. Uh I would like never recommend using like these models past like 200K tokens, under 100K if you can. Um the the the intelligence uh just kind of like falls off a cliff at some point. Um sorry, Anthropic if you're if you're watching. But I I think that uh you know, compaction is like a very useful tool if you are going to have to take a cache miss anyway um one way or another, like when you're routing to another model and you want to just like minimize the the window there.
Comments
Sign in to comment.
Loading comments…