We doubled LFM2.5-8B-A1B's tokenizer from 65K to 128K to fix the languages it split too finely. Today we're sharing the recipe for upgrading a pretrained model's tokenizer in place.
> Thai now takes 4.0× fewer tokens, Vietnamese 2.6×, Hindi 2.4×
> Est. 2.2 to 3.7× faster per-character decoding on-device for these languages
🧵
On-device, at batch size 1, the LM-head reads the whole vocabulary on every token, so a bigger vocabulary is a bigger matrix to stream and to keep in RAM. Edge models ship compact vocabularies and live with the resulting fragmentation.
(2/5)
The approach: extend the tokenizer in place. Continue its existing BPE merges, initialize new tokens from what the model already knows, then a two-stage adaptation. The full method, the ablations, and two reasonable-looking choices that broke the model are in the technical report.
(3/5)
The result: compression where it’s needed without sacrificing quality. English and code stay at parity.
Global-MMLU: Vietnamese +11.6, Indonesian +9.1, Hindi +7.6. Previously supported languages hold steady.
(4/5)
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Why it matters
Vocabulary growth normally means retraining from scratch. This reuses the existing run, cuts tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → counts 2.4x to 4x for underserved languages, and the report names two plausible choices that broke the model.
Read the full thread on X ↗
