LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
Why it matters
Most coding-tool advice stops at prompts; this goes a layer down to explain why your GPU sits idle burning most of its FLOPs during inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → — it's starved for memory bandwidth, not compute. Turc uses the roofline model to make KV cacheThe memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.Full definition →, speculative decodingA speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.Full definition →, and diffusion LLMs click as engineering responses to the same bottleneck. Watch it if you want to reason about latency, batching, and hardware choices instead of guessing.
Key quotes
“Yet it's sitting idle, waiting for parameters to arrive.”
Julia Turc
“for every single token generated, the entire model, all billions of parameters, must complete this journey from the HBM to the processor”
Julia Turc
“Option one, just wait for NVIDIA to improve memory bandwidth, but that's not very high agency.”
Julia Turc
“We are reaching a point where system level engineering optimizations have picked most of the low hanging fruit.”
Julia Turc
“Diffusion LLMs are a completely new paradigm. They no longer generate one token at a time, but rather one draft at a time.”
