MergeKit: A Toolkit for Merging Large Language Models

About

MergeKit is Arcee's open-source library for combining the parameters of existing language models into one model without additional training. It supports merge methods such as linear averaging, SLERP, Task Arithmetic, TIES, DARE and Model Breadcrumbs, plus layer stacking and a Mixture-of-Experts builder. Merges are defined in YAML files and can run on CPU or on a GPU with as little as 8 GB of VRAM, which lets people merge models without a cluster.