Intel / article

A Study on the Impact of Natural Language Differences in Prompts on Automatic Code Generation Using LLMs

Source: arxiv.org
Author: Haruka Tokumasu, Masanari Kondo, Alexander Serebrenik, Dong Wang, Kei Koyanagi, Kotaro Noguchi, Naoyasu Ubayashi, Yasutaka Kamei
Date: Published Sep 17, 2026

Context

A new study tested whether the natural language of a coding prompt, not just its content, changes how often a returns correct code (arxiv.org). The authors took the same programming problems from three benchmarks, AtCoder, LeetCode, and BigCodeBench, translated each into English, Japanese, and Chinese, and ran seven models, including GPT-4o, DeepSeek-V3.2, and two sizes of Qwen2.5-Coder, against every version, checking whether the generated code passed the 's test cases. Accuracy was consistently highest when a problem was presented in the language each benchmark was officially built around, meaning the wording language itself, independent of the underlying problem, shifts how much working code comes back. Translating a prompt into that preferred language recovered some of the gap, but the paper reports the effect was inconsistent across datasets and models, and smaller for benchmarks whose problem statements were longer and more narrative in style.

Key takeaways · AI-distilled

Each dataset's officially supported language (the language its problem statements were originally written in) produced the highest median accuracy across the seven models tested — accuracy dropped when the same problems were translated into a different language.
Translating problems back into the dataset's native language before prompting improved accuracy on average, but the improvement was inconsistent — it did not reliably help across every dataset and model combination.
AtCoder problems had an unusually high share of narrative-style and longer problem statements compared to LeetCode and BigCodeBench, and showed a stronger language-bias effect — pointing to narrative framing and prompt length, not just language choice, as contributing factors.

Terms in this piece · Glossary

LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.

Read the source arxiv.org