Bpe Stays On Script Structured Encoding For Robust Multilingual Pretokenization 2025 05 30
Source
cohere.com
Author
Cohere
Date
Context
The way a LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → breaks text into units can impose extra overhead on some writing systems. SCRIPT-BPE, described in a research abstract hosted by Cohere, starts from Unicode writing-system and character-category information instead of first converting text into UTF-8 bytes.
The method uses those properties to respect boundaries between scripts, alongside a merging rule that preserves complete characters. The authors report competitive compression while eliminating the encoding-based penalties they identify for non-Latin scripts. This addresses how efficiently text is represented, rather than establishing better language understanding.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
postCohere releases open-weights translation model North Small Translate
articleThe Data Provenance Initiative A Large Scale Audit Of Dataset Licensing And Attribution In Ai 2023 10 25
