Snapkv Llm Knows What You Are Looking For Before Generation 2024 04 22

SnapKV reduces the memory a language model needs to hold a long prompt while it writes an answer. The model keeps a key-value (KV) cache, stored attention information about earlier input, and the abstract says this cache grows with input length, straining memory and time. SnapKV finds that each attention head (one of the model's parallel attention components) consistently focuses on particular prompt positions, a pattern visible in an observation window at the end of the prompt. It then compresses the cache by selecting clustered important positions for each head, without fine-tuning the model. Publisher abstract.

The authors' abstract, hosted by Cohere, reports a 3.6 times increase in generation speed and an 8.2 times improvement in memory efficiency over the baseline at 16,000 input tokens, with comparable performance across 16 long-sequence datasets. These are results under the authors' experimental conditions, supporting selective cache compression as an efficiency technique rather than establishing that every task can discard the same information safely. This explanation rests on the abstract only, not a review of the full paper. Published abstract.