Transplanting tokenizers between models without retraining

Every language model is built around a fixed vocabulary, the list of text fragments ('tokens') it was trained to recognize, and different model families almost never share the same one. That mismatch is normally why merging two models' weights, distilling one into another, or using a small model to speed up a larger one through speculative decoding is so difficult, as Arcee researchers Charles Goddard and Fernando Fernandes Neto describe it, since token identifiers do not correspond to the same text across models.

Their method, detailed in an accompanying blog post, uses Orthogonal Matching Pursuit to find which tokens in two different vocabularies represent similar concepts, then reconstructs one model's missing vocabulary entries as combinations of the other's, without retraining the underlying model. Arcee's post reports three separate results for the technique rather than one combined test: a transplant between Llama and Mistral NeMo that preserved 96% of the original model's performance on language understanding tasks, a case where the technique bridged models whose vocabularies overlapped by only 54%, and transplants completing in under two minutes rather than the hours, days, or added compute a retraining-based approach would require. The post does not state whether the 54%-overlap and under-two-minute figures came from that same Llama-to-Mistral NeMo transplant or a different one.