← All IntelClip / AI ToolsAgentic workloads reshape the inference stack
From MiniMax's Open Weight Model Strategy and Inference Stack · ≈9:09
Shifting from chat turns to hundreds of multi-turn tool calls changes KV-cache strategy, routing and which parts of the stack are worth optimizing.
What’s in it
- Shifting from chat turns to hundreds of multi-turn tool calls changes KV-cache strategy, routing and which parts of the stack are worth optimizing.
Clip transcript
work. So, it's very real. One aspect that I wanted to touch on with this is we're seeing the workloads shift. We're going from kind of predominantly chat workloads where you have turns coming in now to agentic workloads where you've got this thing sitting inside a harness and you're doing hundreds and hundreds of multi-turn tool calls. >> Yeah. >> Does that change the way you build the inference stack? >> Yeah, it definitely does. So, these agentic turn-based workloads, they go into everything from informing your KV cash, your prompting, your pieces like this. So, it it informs what part of the stack you want to go optimize because now I think when we're in the chat chat chat world, you have a system prompt of a few thousand, and then you just have the the chat logs. Now, with the coding base agentic workflows, you'll upload your whole code base to the model, and that's a very different optimization and routing and kernel challenge than than just the the chat base workload. So, yeah, we're we we follow these work codes very closely. It's really interesting to see how they evolve and and how to adapt the inference stack and the inference engines to to really take to really serve them well.
Comments
Checking sign-in…
Loading comments…