Vibeleaderboard
← All Intel
Intel / article

Fast Transformer Decoding: One Write-Head is All You Need

Source
arxiv.org
Author
Noam Shazeer
Date

Context

Multi-query reduces the amount of data a model repeatedly reads while generating text. Conventional multi-head attention maintains separate keys and values, the representations its heads consult. The 2019 proposal shares those keys and values across the different heads, shrinking the stored data and reducing the memory bandwidth needed for incremental decoding. The experiments found faster generation with a small decline in quality relative to the baseline. The tradeoff is explicit: sharing more of the attention data makes generation cheaper, but can change the model's quality.

Terms in this piece · Glossary
  • attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
  • KV cache — The memory a model keeps about text it has already read, so generating each new token doesn't require reprocessing the whole conversation.
Recommended reads
Comments

Checking sign-in…

Loading comments…