Why LLM Inference Is Memory-Bound (Julia Turc)
- Source
- youtube.com
- Author
- Julia TurcTop Viber
- Date
Context
A powerful GPU can spend much of its time waiting for model data. In Julia Turc's explanation, generating one piece of text at a time repeatedly streams the model's learned weights from GPU memory into the processor. The video's simplified example uses 16 gigabytes of weights and 3.2 terabytes per second of memory bandwidth: about 200 complete transfers per second, even before accounting for computation.
That is an illustrative limit for the single-response setup, not a universal speed ceiling. Turc explains how batching serves several requests with the same weight transfer, increasing the useful work performed per byte moved, although the memory each request needs for its earlier limits how far batching can go.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
- KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
- speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
“Yet it's sitting idle, waiting for parameters to arrive.”
Julia Turc
“for every single token generated, the entire model, all billions of parameters, must complete this journey from the HBM to the processor”
Julia Turc
“Option one, just wait for NVIDIA to improve memory bandwidth, but that's not very high agency.”
Julia Turc
“We are reaching a point where system level engineering optimizations have picked most of the low hanging fruit.”
Julia Turc
“Diffusion LLMs are a completely new paradigm. They no longer generate one token at a time, but rather one draft at a time.”
Julia Turc
Checking sign-in…
Loading comments…