SWORD: Wikidata-based Distortions Reveal Hidden Cross-Lingual Inconsistencies in LLM Factual Error Rejection

A model that answers factual questions similarly well in several languages may still differ sharply in its ability to reject false statements. SWORD tests that ability by modifying facts from Wikidata into grammatically valid but incorrect statements in eight languages. The researchers report cross-language gaps reaching 28 percentage points in some models, with particular weaknesses in East Asian languages despite comparable baseline accuracy.

This changes what a multilingual evaluation needs to establish. Selecting a correct answer and recognizing an incorrect premise test different behavior. An aggregate score can conceal a model's difficulty identifying false information in a particular language.

“Modern LLMs demonstrate impressive multilingual performance, yet standard benchmarks primarily reward selecting correct answers rather than evaluating genuine factual understanding.”

“models counterintuitively achieve higher accuracy on semantically plausible distortions than on nonsensical random substitutions, suggesting reliance on distributional familiarity rather than genuine factual verification.”

“cross-lingual performance gaps reaching up to 28 percentage points (49\% relative reduction) in some models.”