Beyond Two Bytes per Letter: Tokenization Overhead in Cyrillic AI Systems
Source
arxiv.org
Author
Ivan Dobrovolskyi
Date
Why it matters
Puts numbers on the tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → tax Cyrillic-script languages pay in spend and usable context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → across production tokenizers, and measures two fixes — prompt compression and a balanced byte-level BPE.
Terms in this piece · Glossary
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
RAG — Retrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
articleNot All Tokens Are Equal: Inflation-Aware Routing for Agentic LLM SystemsHeming Fu, Shan Lin, Qianqian Xie, Guojun Xiong
articleThe Functionalizer: Lossless Functional Decomposition for Subword TokenizationConnor Makowski, Willem Guter
articleDoes a Language Server Save Tokens for Coding Agents? A Measurement Methodology and Preliminary StudyPengcheng Xu