Each dataset's officially supported language (the language its problem statements were originally written in) produced the highest median accuracy across the seven models tested — accuracy dropped when the same problems were translated into a different language.
Translating problems back into the dataset's native language before prompting improved accuracy on average, but the improvement was inconsistent — it did not reliably help across every dataset and model combination.
AtCoder problems had an unusually high share of narrative-style and longer problem statements compared to LeetCode and BigCodeBench, and showed a stronger language-bias effect — pointing to narrative framing and prompt length, not just language choice, as contributing factors.
Why it matters
If your team prompts coding models in a non-English language, this quantifies the accuracy cost of that choice across multiple model families and benchmarks.
articleDo Influence Tactics Matter? Investigating Prompt Framing Effects in LLM Code GenerationAlex Deaconu, Anubhav Gupta, Manaal Basha, Nicholas Haydu, Gema Rodr\'iguez-P\'erez
articleAn Extensive Empirical Study on Code Translation TechniqueRuihang Fan, Jiajun Jiang, Xinpeng Wang, Jiateng Fu, Fengjie Li, Jiasi Shen
articleCompound Prompt Constraints in LLM Code Generation: A Factorial Study of Format, Persona, and UrgencyShrenik Jadhav, Nickalsa LaPlaca, Caleb Stone, Ashok Raja, Omar Ochoa, Vidhyashree Nagaraju