How Liquid AI doubled a shipped model's tokenizer without retraining
- Source
- x.com
- Date
We doubled LFM2.5-8B-A1B's tokenizer from 65K to 128K to fix the languages it split too finely. Today we're sharing the recipe for upgrading a pretrained model's tokenizer in place. > Thai now takes 4.0× fewer tokens, Vietnamese 2.6×, Hindi 2.4× > Est. 2.2 to 3.7× faster per-character decoding on-device for these languages 🧵
On-device, at batch size 1, the LM-head reads the whole vocabulary on every token, so a bigger vocabulary is a bigger matrix to stream and to keep in RAM. Edge models ship compact vocabularies and live with the resulting fragmentation. (2/5)
The approach: extend the tokenizer in place. Continue its existing BPE merges, initialize new tokens from what the model already knows, then a two-stage adaptation. The full method, the ablations, and two reasonable-looking choices that broke the model are in the technical report. (3/5)
The result: compression where it’s needed without sacrificing quality. English and code stay at parity. Global-MMLU: Vietnamese +11.6, Indonesian +9.1, Hindi +7.6. Previously supported languages hold steady. (4/5)
Context
Liquid AI expanded the vocabulary of LFM2.5-8B-A1B so languages such as Thai require fewer text units, or , to express the same content. Its tokenizer grew from roughly 65,000 entries to 128,000. The company reports that Thai text uses about one-quarter as many tokens, reducing the number of decoding steps needed to produce it. Method and results.
The upgrade reused the pretrained model but still required training: first the new vocabulary entries, then the whole model. A larger vocabulary makes each decoding step somewhat slower, so the benefit depends on how much text compression improves. The reported 2.2–3.7× speedup for targeted languages is an estimate of output speed per character.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
postLiquid AI ships LFM2.5-24B-A2B open weights for local agents
postToday we release Open d1: two open-weight multimodal models in our d1 decision…
postLiquid AI's d1 decision model adds vision at a fraction of frontier cost
articled1 is now available on @vercel AI gateway!We worked with the amazing vercel…
Checking sign-in…
Loading comments…