Vibeleaderboard
← All Intel
Clip / Other

Self-hosting changes the cache/cost economics fundamentally

From The State of Model Routing — NVIDIA, Cognition, OpenRouter · ≈34:57

“If you self-host, you can optimize specifically for your use, and you'll likely pay much less.”

“The 5-minute uh window is what a lot of providers right now put, but that's uh that's that's more an operational operational operational determination rather than a like a science-based or like a core physics law determination.”

What’s in it
  • Explains why the common 5-minute cache window is a business choice, not physics
  • Shows how self-hosting an LLM lets you tune caching to your own workload
  • Breaks down why API providers price by averaging everyone's usage pattern
Clip transcript
durations rather than just the 5-minute, 1-hour? >> It depends on who's do who's deploying the model where, right? So, if you have a GPU which has like a lot of memory uh which like the ratio of let's say SCMs to memory is memory more heavily skewed or if you're working with unified memory uh and you have systems like where I Rubin, uh you you have a lot of tricks to play here, right? Uh the 5-minute uh window is what a lot of providers right now put, but that's uh that's that's more an operational operational operational determination rather than a like a science-based or like a core physics law determination. So, you can technically see over time maybe some uh some APIs are priced differently, uh but uh if you do self-deploy again, you can you can get past a lot of this. The the cost economics really change when you move from self-hosted models to uh API providers uh because you have a lot more control and uh you don't have to guess the shape of your workload. So, let's say if your workload is 32K uh like on average 32K cache, 1K input, 1K output, uh and someone else's like let's say 64K uh 1K 1K, uh if you use some provider, they are amortizing everyone's use uh use case and then giving you a price, right? And they have optimized, quote and quote, for general use. If you self-host, you can optimize specifically for your use, and you'll likely pay much less.
Recommended reads
Comments

Checking sign-in…

Loading comments…