Understanding And Mitigating Language Confusion In Llms 2024 06 28
Source
cohere.com
Author
Cohere
Date
Context
A multilingual model can answer in a different language than the one the user wanted. Cohere's 2024 research abstract reports this failure across a benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → covering 15 languages, with more confusion under complex prompts and higher sampling temperatures, which make generation more random.
The benchmark repository distinguishes answering in the language of the question from following an instruction to answer in another language. It checks whether output lines match the requested language and separately detects certain unwanted English words. This makes language compliance a specific, testable requirement for multilingual applications, alongside the quality of the answer itself.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
postCohere releases open-weights translation model North Small Translate
articleThe Data Provenance Initiative A Large Scale Audit Of Dataset Licensing And Attribution In Ai 2023 10 25
articleA Systematic Evaluation of Cross-Lingual Consistency Enhancement Methods in Multilingual Language ModelsJirui Qi, Mingyang Wang, Hinrich Sch\"utze, Raquel Fern\'andez, Arianna Bisazza
articleMeasuring the Cross-Lingual Comprehension Gap: How the language of the evidence shapes what language models understandRafael da Silva, Jeff Eicher
articleInclude Evaluating Multilingual Language Understanding With Regional Knowledge 2024 11 29Cohere