Intel / article

The Indic Token Tax in 2026: Measuring Tokenizer. Disparity and Its Cost for the Languages of India

Source: arxiv.org
Author: Johin Johny Arimbur, Ankur Ashit Ghosh, Shervin Sam
Date: Published Nov 11, 2026

Context

The same sentence can cost several times more to process in an Indian language than in English, because of how models cut text into pieces. A tokenizer splits text into , the units a model reads and commercial APIs bill by, and this paper from three researchers at Havon AI in Mumbai counts how many tokens nine public tokenizers need for the same content in English and in the languages of India. The test set is FLORES-200, 2,009 professionally translated sentences aligned with English, in all 25 Indian languages it covers, plus 13,732 Hinglish-English social media pairs. On the tokenizer used by GPT-4o, the same content takes 1.57 times the English token count in Hindi, 5.01 times in Odia, and 13.7 times in Santali written in Ol Chiki script. The older GPT-4 tokenizer needs up to 12.8 times. Claude, Gemini, Llama 3 and Gemma 3 were left out because their tokenizers are not public or access is gated.

At an equal price per token, that ratio is the cost multiplier, and a fixed holds proportionally less text. The authors illustrate it with a hypothetical support bot handling 100,000 chats a month on GPT-4o at list prices accessed on 5 November 2026: about INR 56,640 a month in English and INR 102,829 in Marathi. They call these estimates rather than measured production costs, since they rest on stated assumptions and ignore fixed system prompts, and they note that the price differences between models are larger than the tax itself. The paper also says speakers of these languages wait longer for answers, but that follows from the token counts; response times were not measured.

A tokenizer from an Indian model maker narrows the gap without closing it. Sarvam, an India-based company whose 30B and 105B models were announced in 2026, has the only third-party tokenizer that stays below 2.6 times English on every language tested, with a median of 1.53. None of the nine splits words along their grammatical parts: on a Hindi and Marathi set of words divided into morphemes, the smallest units that carry meaning, no tokenizer places more than 35% of its splits inside words on a morpheme boundary. The authors say a companion paper presents a tokenizer that reduces the tax, so this one only measures existing tokenizers. An earlier study of Ukrainian found the same kind of overhead for Cyrillic-script languages.

Key takeaways · AI-distilled

On GPT-4o's tokenizer, the authors find the same content takes 1.57 times the English token count in Hindi, 5.01 times in Odia, and 13.7 times in Santali written in Ol Chiki, measured across all 25 Indian languages in FLORES-200.
Of the nine public tokenizers tested, Sarvam-30B/105B is the only third-party one that stays under 2.6 times English on every language (median 1.53). GPT-4's tokenizer needs up to 12.8 times.
In the paper's illustrative scenario using list prices from Nov 5, 2026, a GPT-4o support bot handling 100,000 chats a month costs about INR 56,640 in English and INR 102,829 in Marathi. The authors label these estimates, not measured production costs.
No tested tokenizer places more than 35% of its word-internal splits on Hindi or Marathi morpheme boundaries (boundary F1 at most 0.48), measured against an openly licensed segmentation set.

Terms in this piece · Glossary

token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.

Read the source arxiv.org