Research Spotlight 3 Learnings From 3 Use Cases Of Mergekit

Saved snapshots from one training run can be averaged to produce a better model. Arcee AI's June 2025 MergeKit overview summarizes a paper on this idea, called Pre-trained Model Average, and says Arcee used the approach while developing its AFM-4.5B model. The overview also reports that other developers have used merging without disclosing the techniques.

The underlying paper reports that merging checkpoints from the stable, constant-learning-rate phase of training produced consistent and significant performance gains across the models it tested. Fine-tuning that started from the averaged weights showed steadier gradient behavior and fewer loss spikes than the baseline, though the paper says gains in final downstream performance were occasional rather than consistent. The practical lesson is that the intermediate checkpoints of a training run have value beyond the last one, as the paper tested it.