Clip / Entertainment

Every token requires a full model stream

From Why LLM Inference Is Memory-Bound (Julia Turc) · ≈3:37

“The important takeaway is that for every single token generated, the entire model, all billions of parameters, must complete this journey from the HBM to the processor.”

“It's a continuous pipeline of bytes being copied.”

What’s in it

Understand why every token requires streaming all model weights from HBM
Learn how autoregressive LLM inference is memory-bandwidth bound
Grasp the 'model stream' mental model for GPU-to-processor data flow

Watch this moment

Clip transcript

For auto regressive LLMs in particular, the result of a forward pass is a single next token prediction. In practice, model weights are copied in smaller chunks than entire layers, but layers are just easier to visualize. The important takeaway is that for every single token generated, the entire model, all billions of parameters, must complete this journey from the HBM to the processor. Throughout the video, I'll refer to this journey as a "model stream" or "model copy". Just remember, it's not a one-time atomic transfer. It's a continuous pipeline of bytes being copied.