Vibeleaderboard
← All Intel
Intel / article

Co-Designing Sparse Attention for Efficient Long-Context Inference

Source
developer.nvidia.com
Author
Timmy Liu
Date

Context

Sparse does not remove the cost of long ; it moves most of it into a component called the indexer, and an NVIDIA technical post measures how to keep that component cheap. In ordinary dense attention, each new token (the query) is compared against every earlier token (the keys), so the cost per token grows with context length. NVIDIA says that at 128,000 tokens attention can take most of the prefill time in a layer, prefill being the pass that reads the prompt before any output is written. Sparse attention trained into the model lets each query attend only to the few thousand most relevant keys. But finding them requires the indexer, a small attention-like block that still scores every earlier key, after which a Top-k step keeps the highest scores. In DeepSeek's V3.2 prefill layer, the indexer and Top-k take 47% of the time.

The authors swept the design choices of a production FP8 indexer kernel, and the clearest result concerns key heads, the separate sets of keys the indexer scores against. During decode, when the model writes output one token at a time, the indexer's runtime is set by reading its stored keys from GPU memory. Each extra key head means another full read, so going from one key head to eight made decode 7.6 times slower, while prefill slowed only 1.3 to 1.6 times. Query heads behave the opposite way: they add arithmetic but no new memory reads, so a 16-fold larger group of query heads changed decode runtime by 2%. The recommendation is one key head, with the scores from all query heads combined into a single score per key, which is what most current designs in NVIDIA's comparison table, including DeepSeek's and GLM 5.2's, already do. The same logic applies to , where the model drafts several tokens and checks them in one step: the indexer scores the extra draft tokens almost for free as long as query heads times query tokens fit within one Tensor Core tile of about 200 rows.

Beyond that, the only way to reduce indexer work is to score fewer keys. Compressing the index so one entry stands for several tokens, indexing hierarchically, or sharing one index across layers all cut work in proportion; halving the compression ratio doubled runtime in prefill and decode. Top-k cost also follows the number of keys scored, not how many are kept. Two kernel techniques help it further: fusing Top-k into the indexer avoids writing every score to memory and reading it back, and reusing the previous decode step's selection as a starting guess made Top-k 2.1 to 2.4 times faster than a standard radix select on DeepSeek V3.2 and V4 decode. For the final attention step, selecting contiguous blocks of keys rather than scattered single tokens lets the kernel fetch each block once and share it across queries. Block size matters: with the same 2,048 selected keys, halving the block size nearly doubled runtime.

These are kernel speed measurements on NVIDIA GPUs, written as guidance for teams designing new models with sparsity built in, not for methods added to an existing model after training. The post does not report how these choices affect a model's accuracy, so they describe what the hardware rewards rather than which design produces the better model. It continues an earlier NVIDIA series that applied the same approach to dense attention.

Key takeaways · AI-distilled
  • Sparse attention does not remove the full-context cost, it moves it: the indexer still scores every key, and NVIDIA measures the indexer plus Top-k at 47% of a DeepSeek V3.2 prefill layer.
  • Because indexer decode is memory-bound on reading the key cache, NVIDIA found runtime moved only 2% as group size grew 16x, so extra query heads and speculative draft tokens are nearly free while they fit in one pass.
  • Adding indexer key heads is costly in decode: each one forces another full read of the key cache, and NVIDIA measured runtime rising 7.6x from one key head to eight, versus only 1.3x to 1.6x in prefill.
  • With a head dimension of 64, prefill is bound by the ReLU and weighted-sum epilogue on CUDA cores while Tensor Cores sit idle, so widening to 128 cost only 1.1x to 1.2x in NVIDIA's measurements.
  • For block-level core attention at a fixed 2,048-key budget, NVIDIA found halving the block size nearly doubles runtime, because each block adds its own query load, partial result and merge step even though FLOPs and bytes stay the same.
Terms in this piece · Glossary
  • attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
  • context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
  • speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
  • inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Read the source developer.nvidia.com
Recommended reads
Comments

Checking sign-in…

Loading comments…