← All IntelClip / AI ToolsKV cache at million-token context is a distributed systems problem
From MiniMax's Open Weight Model Strategy and Inference Stack · ≈16:23
Frames long-context serving as storage/lookup/transfer engineering — where the cache lives, whether you've seen it before, how you fetch and move it.
What’s in it
- Frames long-context serving as storage/lookup/transfer engineering — where the cache lives, whether you've seen it before, how you fetch and move it.
Clip transcript
you, Dan, for is KB cache. So, if let's say you you have concurrent requests that are 500 to a million thousand context length long, how do you deal with the KB cache that just keeps on growing, and how does the infrastructure deal with that? >> Yeah, so there there's a lot of different pieces that that you put there. Like in in some sense, it's like recreating a distributed file system. So, we we're in some sense building something like that or a very big database. It's it's pretty simple in theory. It's like the type of thing that you should have done in your third year of undergrad or something like that, but most of us actually skipped that class, so now we're rediscovering it live in in industry. But it's it's all about where do you store that cache? How do you know Have you have you seen this before? How do you fetch it? How do how do you send it from one place to another? So, yeah, it's it's a it's it's not that complicated, but you you do have to make sure that that you do a good job.
Comments
Checking sign-in…
Loading comments…