Distilling Kimi Delta Attention Into Afm 4 5b And The Tool We Used To Do It

Arcee AI tested whether an existing language model could acquire a different attention architecture without being pretrained from scratch. In a December 2025 experiment, the author replaced many of AFM-4.5B's attention layers with Kimi Delta Attention, an architecture from Moonshot AI, copied most of the original weights across, and trained the altered model to imitate the original, a process called distillation.

Under this small training budget (about a billion tokens), a hybrid that kept some full-attention layers recovered dramatically more performance than the version using only Kimi Delta Attention, with the gap larger on math than on knowledge benchmarks. The original model still performed best. The author cautions that this does not show the new architecture is inherently worse at math, since the compute was small and the math benchmark is sensitive to training-data formatting. The result suggests the mix of retained and replaced layers matters in this setup, not that conversion works in general.