Prefix Sliding: A Cheaper Alternative to Full Attention for Long Reasoning
- Source
- x.com
- Date
Long reasoning traces are expensive because full attention grows with context length. @Muennighoff et al find that maybe LLMs can stand to forget a little… with Prefix Sliding! https://t.co/SFpPUjbycB
Context
Long reasoning traces are expensive because full grows with : each new token is compared against everything before it. Stanford AI Lab points to a paper on Prefix Sliding, a method built on the idea that a model can stand to forget a little.
According to the paper, Prefix Sliding keeps a fixed prefix plus a sliding window of recent tokens in memory instead of the whole trace. Applied to an existing model with no additional training, it reports close to a 3x speedup with performance held roughly steady. It also reports that training a model with Prefix Sliding using reinforcement learning lets reasoning scale past a hundred thousand tokens.
These are the paper's reported results for two settings, training-free and RL training, not a claim that the method suits every reasoning workload.
- attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
- context compaction — Summarizing an agent's earlier conversation to free room in the context window so a long session can keep going.
Checking sign-in…
Loading comments…