RBS-Attention: Radius-Bounded Sparse Prefill for Long-Context Large Language Models
Source
Chuxu Song, Jiuqi Wei, Zhencan Peng
Author
Chuxu Song, Jiuqi Wei, Zhencan Peng
Date
Key takeaways · AI-distilled
The failure mode it targets, mean dilution, is specific: averaging a block's tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → into one centroid can hide a highly relevant token among many irrelevant ones, causing sparse block-selection to miss it.
Its rescue branch uses each key-block's maximum radius, and its prompt-, layer-, and head-dependent distribution, to flag blocks at risk of being underestimated by the centroid branch alone, then thresholds the two branches independently before combining their masks.
At 128K context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → on H100s with Qwen3-30B-A3B-Instruct-2507-FP8, it reaches 20.65x standalone prefill-attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition → speedup, 11.92x vLLM prefill-attention speedup, and 5.97x end-to-end time-to-first-token speedup - three distinct numbers, not one blended figure.
On dense Qwen3-32B, it holds 88.65 overall RULER accuracy versus 89.52 for full dense attention, a small measured quality cost alongside the prefill speedup.
Terms in this piece · Glossary
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
A drop-in sparse-attention technique that preserves standard block-sparse FlashAttention execution could cut prefill latency substantially for long-context AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → workloads without retraining.
