Fishing For Magikarp Automatically Detecting Under Trained Tokens In Large Language Models 2024 09 24
Source
Cohere
Author
Cohere
Date
Terms in this piece · Glossary
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Why it matters
Under-trained tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → make models behave erratically on rare inputs. Automatic detection means a tokenizer's vocabulary can be audited up front instead of glitch tokens surfacing in production.
articleLanguage Models Don T Always Say What They Think Unfaithful Explanations In Chain Of Thought Prompting 2023 05 07
articleMteb Massive Text Embedding Benchmark 2023 03 19
articleConsent In Crisis The Rapid Decline Of The Ai Data Commons 2024 07 19
articleUnderstanding Likelihood Over Optimisation In Direct Alignment Algorithms 2024 10 18
articleFishing For Magikarp Automatically Detecting Under Trained Tokens In Large Language Models 2024 05 08Cohere
articleBreaking Down Model Vocabulary Barriers With Tokenizer TransplantationArcee AI
articleOne Tokenizer To Rule Them All Emergent Language Plasticity Via Multilingual Tokenizers 2025 05 30Cohere