Fast Transformer Decoding: One Write-Head is All You Need
Source
arxiv.org
Author
Noam Shazeer
Date
Why it matters
It named the real bottleneck in generation: not the arithmetic, but the size of the KV cacheThe memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.Full definition → carried per tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition →. Sharing one KV head across all query heads shrinks that cache sharply, which is what makes long contexts and large batches affordable.
Terms in this piece · Glossary
KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
