Fishing For Magikarp Automatically Detecting Under Trained Tokens In Large Language Models 2024 05 08
Source
Cohere editorial sitemap
Author
Cohere editorial sitemap
Date
Terms in this piece · Glossary
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Why it matters
tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → present in the tokenizer but barely seen during training make models behave erratically when they appear in input. This detects them automatically in any model, which matters for robustness and for anything processing untrusted text.
articleCommit0 Library Generation From Scratch 2024 12 02
articleLlmcrit Teaching Large Language Models To Use Criteria 2024 03 02
articleSeacrowd A Multilingual Multimodal Data Hub And Benchmark Suite For Southeast Asian Languages 2024 06 14
articleBpe Stays On Script Structured Encoding For Robust Multilingual Pretokenization 2025 05 30
