context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
Why it matters
context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → extension usually wrecks short-context scores. This walks the actual experiments used to buy 64k without paying that tax, including merging, restoration distillationTraining a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.Full definition →, and weight soups.
