Linear-Time Context Compression with State-Space Encoding
Source
arxiv.org
Author
Marc Serra Ortega, Andrey Barsky, Carlos Boned Riera, Tom\'as Ockier, Dimosthenis Karatzas
Date
Why it matters
Soft-prompt context compactionSummarizing an agent's earlier conversation to free room in the context window so a long session can keep going.Full definition → has needed an expensive Transformer pass, which limited it to reusable contexts. A linear-time SSM encoder makes compression workable online for a frozen LLM.
Key takeaways · AI-distilled
The Boa Compressor uses a Mamba state-space model to compress arbitrary-length context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → in one linear-time pass into a representation that conditions a frozen target LLM.
Boa is trained by distillationTraining a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.Full definition →: the student is the same LLM reading the compressed representation, matched to the next-token distribution of the LLM reading the full context.
The authors report Boa matches or beats state-of-the-art offline compressors at equal compression ratios while running an order of magnitude faster than transformerThe neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.Full definition →-based ones, making per-query compression practical, and it scales to 16x compression.
Terms in this piece · Glossary
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
context compaction — Summarizing an agent's earlier conversation to free room in the context window so a long session can keep going.
transformer — The neural network architecture behind modern AI models, built on attention — letting every word directly consider every other word in parallel.
