TriusLD – A Small Tokenizer for LLMs with Robust Multilingual Support

Shows how to allocate a fixed vocabulary budget across scripts so multilingual text costs fewer tokens without badly hurting English. Useful for anyone training small models or judging tokenizer cost.

• token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.

• articleBeyond Two Bytes per Letter: Tokenization Overhead in Cyrillic AI SystemsIvan Dobrovolskyi
• postHow Liquid AI doubled a shipped model's tokenizer without retrainingLiquid AI
• articleBpe Stays On Script Structured Encoding For Robust Multilingual Pretokenization 2025 05 30Cohere

Checking sign-in…

Loading comments…