How Specializing the Vocabulary of SLMs Can Cut Inference Cost by 77% in Real Applications
- Source
- huggingface.co
- Date
Context
Dharma-AI runs an automated grader for Portuguese essays, built on Qwen3-4B. In a Hugging Face post previewing a paper still under review, the team argues that a fine-tuned model's inherited tokenizer can be one of its largest costs. The original Qwen3-4B vocabulary has 151,669 entries, built for many languages. That vocabulary caused two problems. It split the grader's most repeated output, the same JSON structure and phrases on every request, into many . It also forced the model to score every entry, including ones the task never uses, at each step of generation.
Their method, VocabX, chooses new tokens by what they cost to serve, not by how often they appear. Generated tokens are memory-bound and expensive. On the grader's setup, the authors measured a generated token at 7.4 times the cost of a token read, so a token saved in the response is weighted 7.4 times one saved in the prompt. VocabX rebuilds sub-word pieces for the domain and also adds macro tokens, each standing for a whole repeated span such as a line or JSON envelope. It caps vocabulary size, which shrank the output layer from about 152,000 rows to about 65,000. That differs from approaches that grow the vocabulary instead. Liquid AI recently described doubling a shipped model's tokenizer to fix fragmented languages, and its in-place method is one of the baselines VocabX is compared against.
The team trained seven graders that differed only in tokenizer and ran each on the same 500 requests, on an RTX 5090 with vLLM. With VocabX, average responses fell from 350 to 101 tokens. Cost, calculated from measured runtime, fell 77.0%, and the bench score rose from 0.703 to 0.810. The most compact version was not the best. Fusing each competency's grade into its surrounding JSON macro dropped quality to 0.689, below the production model. Giving the grade its own token restored quality and was slightly cheaper in their measurement. The authors' lesson is that compression should not swallow the decision the model exists to make. They report that VocabX also compressed more than the strongest reference method in all fifteen domain and model-family combinations they tested, but the full tables will only appear with the paper.
- Dharma-AI rebuilt the tokenizer of a fine-tuned Qwen3-4B Portuguese essay grader, keeping base model, data and recipe fixed. Average response length fell from 350 to 101 tokens and measured cost fell 77%, while its bench score rose from 0.703 to 0.810.
- VocabX picks tokens by serving cost, not frequency: each saved token is weighted by where it appears. In their grader a generated token cost 7.4x an input token, because decode is memory-bound, so shrinking output is worth far more than shrinking input.
- Besides sub-word merges, it adds macro tokens for whole repeated spans such as JSON envelopes, found by five miners. The output layer shrank from about 152,000 to 65,000 rows, which also cuts per-token work since the LM head reads every row at each step.
- Fusing each grade with its surrounding JSON into one macro dropped quality to 0.689. Giving the grade its own token cost two extra tokens yet scored 0.810 and was cheaper, so compression should not swallow the decision the model exists to make.
- Across five public domains and three model families, VocabX compressed more than FVT, multi-word tokenization and in-place BPE in all 15 pairings, by 1.2-9.3 points. The figures come from a paper still under review and may change.
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
Checking sign-in…
Loading comments…