One Tokenizer To Rule Them All Emergent Language Plasticity Via Multilingual Tokenizers 2025 05 30

A model's ability to learn another language later can depend on how its text vocabulary was designed at the start. Research presented by Cohere tested a tokenizer, the component that divides text into model-readable units, trained on more languages than the model's main pretraining covered.

The publisher abstract reports better adaptation to additional languages than with tokenizers limited to the original training languages, with little performance compromise on most original languages. The result supports preparing the text representation for future expansion; later language training still does the work of adapting the model.