Include Evaluating Multilingual Language Understanding With Regional Knowledge 2024 11 29

INCLUDE evaluates multilingual AI using questions drawn from local exams. Cohere's research page for the paper describes a collection of 197,243 question-answer pairs across 44 written languages. Its abstract identifies a weakness in the common practice of translating English benchmarks: translation changes the language while ignoring the regional and cultural knowledge of the places where a system would actually be used.

Locally sourced questions give evaluators a way to test knowledge and reasoning relevant to different settings, which helps separate the ability to answer translated questions from performance on material that originates in the communities a system is meant to serve. The contribution is an evaluation resource, with exam-based coverage defining its scope; the abstract reports no model rankings.