Last week, we launched AFM-4.5B, our first foundation model.
In this post by @chargoddard , you will learn how we extended the context length of AFM-4.5B from 4k to 64k context through aggressive experimentation, model merging, distillation, and a concerning amount of soup.
Bon appétit 😋
Blog post:
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
Why it matters
Extending context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → by merging and averaging checkpoints is far cheaper than retraining for long context, and this documents what the combination actually achieved on a real 4.5B model.
