Different models have different vocabularies, making it difficult to efficiently combine them for merging, distillation, or speculative decoding
In this new paper, @arcee_ai researchers Charles Goddard and Fernando Fernandes Neto introduce a revolutionary approach called "tokenizer transplantation," utilizing a technique known as Orthogonal Matching Pursuit (OMP).
Think of it as a sophisticated translation system that can convert between different model vocabularies without any retraining. Here's the key insight: even though different models use different vocabularies, the concepts they represent often align in predictable ways. Our method finds these alignments and uses them to transplant one model's vocabulary into another.
If you'd like to learn more, please read our high-level blog post (https://t.co/iUbXxjTVsV), or dive into the research paper (https://t.co/sQHHBjNt8a).
Learn more about model merging and get expert support at https://t.co/ut82Ns22GE.
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.
speculative decoding — A speed trick where a small model drafts several tokens ahead and the big model verifies them in one pass, often doubling generation speed.
Why it matters
Mismatched tokenizers are the usual blocker for merging or distilling across model families, or for using one model as a speculative decoder for another. This offers a training-free way to align vocabularies.
