Mergeable Model-Side Aggregation States for Long-Context Language Models
Source
Dachuan Song, Junyu Yin, Zechen Hu, Xuan Wang
Author
Dachuan Song, Junyu Yin, Zechen Hu, Xuan Wang
Date
Terms in this piece · Glossary
attention — The mechanism that lets a model weigh which earlier words matter for the word it's currently processing — the core operation of a transformer.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Why it matters
Long-context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → models get distinct counts and group-bys wrong as context grows. A 2 KiB HyperLogLog sketch maintained beside a frozen model held 1.6% error over a million records and 99.2% accuracy on aggregate-then-reason tasks, with segment states merging exactly.
