How AFM-4.5B's context window went from 4k to 64k
- Source
- x.com
- Date
Last week, we launched AFM-4.5B, our first foundation model. In this post by @chargoddard , you will learn how we extended the context length of AFM-4.5B from 4k to 64k context through aggressive experimentation, model merging, distillation, and a concerning amount of soup. Bon appétit 😋 Blog post:
Context
A week after launching AFM-4.5B, Arcee's Charles Goddard published a write-up of extending the model's usable , the amount of text it can process in a single prompt, from 4,000 to 64,000 .
Per the linked blog post, the approach combined YaRN scaling of the model's positional with training directly at 64,000-token sequence length, then used linear merging, averaging weights from several separately trained checkpoints, to recover performance lost during those experiments, alongside short-context that trained the long-context model against the original short-context model as a teacher. Arcee also found that deliberately overshooting the context target during training and then reducing the scaling afterward improved results.
Arcee reports the final merged model scored 65.0% on MMLU and 45.0% on BBH, close to its pre-extension baseline of 66.0% and 45.2%, while retaining 100% needle-in-a-haystack recall at 32,000 tokens and 97% at 64,000 tokens, indicating the long-context gain came with only a small measured cost to short-context accuracy on those two benchmarks.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
- distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
Checking sign-in…
Loading comments…