Extending Afm 4 5b To 64k Context Length
- Source
- arcee.ai
- Author
- Arcee AI
- Date
Context
Giving a more reading space can weaken its performance on shorter tasks. In its AFM-4.5B experiments, Arcee AI expanded the from roughly 4,000 to 64,000 , while repeatedly checking what the model lost along the way.
The team combined long-sequence training with merging model versions and teaching the extended model to reproduce the original model’s short-context behavior. On the two short-context benchmarks Arcee tracked, the final experimental model scored 65.0% on MMLU (against 66.0% for the starting checkpoint) and 45.0% on Big Bench Hard (against 45.2%), so it recovered most but not all of the loss. The reported results concern a preliminary checkpoint that was still being trained: the useful lesson is that extending context required recovering existing abilities as well as learning to handle longer inputs.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Checking sign-in…
Loading comments…