Long reasoning traces are expensive because full attention grows with context length.
@Muennighoff et al find that maybe LLMs can stand to forget a little… with Prefix Sliding! https://t.co/SFpPUjbycB
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
context compaction — Summarizing an agent's earlier conversation to free room in the context window so a long session can keep going.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Why it matters
Prefix Sliding lets models drop earlier reasoning context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → selectively during long inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → chains, avoiding both the memory blowup of full attentionThe mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.Full definition → and the accuracy loss of context compactionSummarizing an agent's earlier conversation to free room in the context window so a long session can keep going.Full definition →, without retraining.
