Intel / article

Tutorial How To Get Started With Mergekit

Source: arcee.ai
Author: Arcee AI editorial sitemap
Date: Published Oct 30, 2026

Context

This April 2024 tutorial by Charles Goddard announced evolutionary model merging in MergeKit, Arcee AI's toolkit for merging existing language models into a single model (Arcee blog). Arcee presents it as filling a gap: Sakana AI had published a paper on evolutionary model merging and its results in March 2024 but, according to Arcee, never released the algorithm behind them. Without automation, Goddard writes, merging is a manual search: trying dozens of merges, evaluating each by hand, and trying to work out how the merge parameters relate to the final model's performance. MergeKit's version lets the user state which qualities the merged model should have and leaves the search for a merge recipe to an optimizer.

Those qualities are expressed as evaluation tasks. MergeKit scores candidates with EleutherAI's evaluation , so any the harness supports can serve as a target, and users can also write custom tasks in YAML. Goddard's example optimizes for spatial reasoning, using a sample of the spartqa-mchoice question set, and for Alpaca prompt formatting, checked by whether the model ends a completed response with an end-of-sequence , because merged models often do not follow any one prompting style. A configuration file then names the models to combine (Hermes 2 Pro Mistral 7B, Dan's Adventurous Winds Mk2 7B and Zephyr 7B beta), the merge method (Task Arithmetic) and how much each task counts. The run keeps merging and scoring candidates, by default until it has evaluated more than 100 merges, then writes the best-scoring configuration to a file that the mergekit-yaml command turns into the final model. Goddard says the minimum is one GPU that can run a model in FP16, that 24GB of video memory is enough for 7B models, and that his demo used a single NVIDIA A100 (MergeKit on GitHub).

Because the optimizer improves only what the tasks measure, choosing the tasks is the main design decision. Goddard deliberately wrote custom tasks instead of optimizing directly against Open LLM Leaderboard scores. The tutorial reports no scores for the merged model it produces, so it shows how to run the search but not how much evolutionary merging improves on a hand-built merge.

Key takeaways · AI-distilled

MergeKit adds evolutionary model merging, a method Sakana.ai showed results for in March 2024 without releasing the algorithm: you define target tasks and an optimizer searches for the merge recipe instead of hand-tuning weights.
MergeKit scores candidate merges with EleutherAI's lm-eval harness, so any lm-eval benchmark or a custom YAML task can be the objective; the tutorial uses quick multiple-choice tasks for spatial reasoning and Alpaca prompt-format compliance.
The tutorial's hardware floor is one GPU that can run the models in FP16, with 24GB of VRAM enough for 7B models; a Ray cluster can spread the work across more GPUs.
By default the search evaluates over 100 merges (adjustable with max-fevals) and writes the best-scoring recipe to best_config.yaml, which mergekit-yaml then turns into the final model.

Terms in this piece · Glossary

LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.

Read the source www.arcee.ai