Breaking Down Model Vocabulary Barriers With Tokenizer Transplantation

Arcee AI describes a method for replacing a model's tokenizer without additional training. A tokenizer defines the text pieces a model processes; replacement requires numerical representations for unfamiliar pieces. The method expresses each new piece using a small combination of pieces shared by both vocabularies, then applies that combination in the receiving model's representation space. The paper's method and results show that this rebuilds the vocabulary embeddings. It preserves much of the tested models' general-task performance, while accuracy on the paper's grade-school math benchmark (GSM8K) fell sharply. The authors hypothesize that this reflects tokenizers that divide numbers differently, such as single digits versus groups of three, and they describe the explanation as supported but not proven.