Intel / article

Understanding and Coding the KV Cache in LLMs from Scratch

Source: magazine.sebastianraschka.com
Author: Sebastian Raschka, PhD
Date: Published Jun 17, 2025

Context

A speeds text generation by keeping intermediate calculations that the model would otherwise repeat. In Sebastian Raschka's walkthrough, the model generates one token at a time. Earlier tokens already have key and value representations used by , so those representations can be stored and reused.

After processing the initial prompt, each new step adds the new token's keys and values to the cache. Attention uses these alongside the stored representations of earlier text. This trades memory for less repeated computation: a longer sequence leaves more information to store, even though reusing it makes generation more efficient.

Key takeaways · AI-distilled

A KV cache stores the key and value vectors already computed for earlier tokens, so each generation step only projects the newest token instead of re-encoding the whole sequence. The saving grows with sequence length.
The cache is -only. Training processes a full sequence in one pass with no autoregressive reuse, so the technique buys nothing during a training run and only costs memory.
The core implementation is two registered buffers on the attention module: concatenate the new keys and values onto them each step, then read the concatenated tensors instead of the freshly computed ones.
Forgetting to clear the cache between generations is a silent bug. The new prompt's queries attend to leftover keys from the previous sequence, so the model conditions on irrelevant and the output goes incoherent.

Terms in this piece · Glossary

KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.

Key quotes

“In short, a KV cache stores intermediate key (K) and value (V) computations for reuse during inference (after training), which results in a substantial speed-up when generating text.”

“So, as we can see, we already get a ~5x speed-up with a small 124 M parameter model and a short 200-token sequence length.”

“Unfortunately, the speed advantages disappear on CUDA devices as this is a tiny model, and the device transfer and communication outweigh the benefits of a KV cache for this small model.”

“Continuously concatenating tensors via torch.cat , as shown earlier, leads to performance bottlenecks due to frequent memory allocation and reallocation.”

Read the source magazine.sebastianraschka.com