Bpe Stays On Script Structured Encoding For Robust Multilingual Pretokenization 2025 05 30
Source
Cohere editorial sitemap
Author
Cohere editorial sitemap
Date
Terms in this piece · Glossary
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
Why it matters
BPE splits non-Latin scripts into many more tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → than English, so the same sentence costs more and degrades further into the context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition →. Handling script boundaries during pretokenization reduces that penalty where it originates.
articleCommit0 Library Generation From Scratch 2024 12 02
articleLlmcrit Teaching Large Language Models To Use Criteria 2024 03 02
articleSeacrowd A Multilingual Multimodal Data Hub And Benchmark Suite For Southeast Asian Languages 2024 06 14
articleFishing For Magikarp Automatically Detecting Under Trained Tokens In Large Language Models 2024 05 08
