Intel / video

Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher

Source: youtube.com
Author: AI Engineer
Date: Published Sep 8, 2026

Context

Serving a language model requires memory for both its learned parameters and the growing records of active conversations. This inference workshop focuses on how longer contexts and more simultaneous requests create pressure on that second category, commonly called the .

The cache stores intermediate calculations used while generating subsequent text. The serving engine vLLM explains that its PagedAttention method divides this cache into blocks allocated as needed, instead of requiring a large continuous reservation for each request. Reducing wasted memory allows more requests to be processed together. That explains why serving software can improve capacity without changing the model's learned parameters. Primary technical explanation.

Terms in this piece · Glossary

inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.

Read the source www.youtube.com