Intel / article

RBS-Attention: Radius-Bounded Sparse Prefill for Long-Context Large Language Models

Source: arxiv.org
Author: Chuxu Song, Jiuqi Wei, Zhencan Peng
Date: Published Sep 21, 2026

Context

RBS- aims to shorten the wait before a long-prompt model starts answering while reducing the risk of skipping useful information. The paper explains that selecting text blocks by their average relevance can hide an important among irrelevant ones. Its additional selection path uses how widely token representations vary within a block to retain blocks that an average could undervalue.

On NVIDIA H100 GPUs, the researchers report a 5.97× time-to-first-token speedup at 128K for Qwen3-30B-A3B-Instruct-2507-FP8, against dense attention under the same setup. This is an approximation with a speed–accuracy tradeoff: tighter selection can discard useful material, so the paper evaluates answer quality alongside speed.

Key takeaways · AI-distilled

The failure mode it targets, mean dilution, is specific: averaging a block's tokens into one centroid can hide a highly relevant token among many irrelevant ones, causing sparse block-selection to miss it.
Its rescue branch uses each key-block's maximum radius, and its prompt-, layer-, and head-dependent distribution, to flag blocks at risk of being underestimated by the centroid branch alone, then thresholds the two branches independently before combining their masks.
At 128K context on H100s with Qwen3-30B-A3B-Instruct-2507-FP8, it reaches 20.65x standalone prefill-attention speedup, 11.92x vLLM prefill-attention speedup, and 5.97x end-to-end time-to-first-token speedup - three distinct numbers, not one blended figure.
On dense Qwen3-32B, it holds 88.65 overall RULER accuracy versus 89.52 for full dense attention, a small measured quality cost alongside the prefill speedup.

Terms in this piece · Glossary

attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.

Read the source arxiv.org