Intel / article

CodeTransBenchmark: Evaluating LLM-based Code Translation and Repair Across Programming Languages

Source: arxiv.org
Author: Vera Kowalczuk, Oliver Wei{\ss}l, Severin Kacianka, Andrea Stocco
Date: Published Sep 18, 2026

Context

Translating code with an AI involves more than requesting equivalent code in another language. CodeTransBenchmark evaluates both the initial translation and subsequent repair, including a step that extracts usable code from inconsistently formatted model responses. Its study covered eight models, three datasets and 12 language pairs. Research abstract.

The authors report that most general-purpose models struggled with the destination language's syntax, while repeated repair using automated feedback improved translation accuracy. This makes the correction process part of the translation system being evaluated: a model's first response and its output after feedback can have materially different success rates. Reported findings.

Key takeaways · AI-distilled

Across 8 models, 3 datasets, and 12 language pairs, models trained specifically for multilingual code (like Codestral) correctly translated the majority of snippets, while most general-purpose LLMs struggled with the target language's syntax rules.
Error analysis found translation accuracy is heavily shaped by the specific pairing of source and target languages and by what each model was trained on, not just overall model quality.
A post-processing step built to tolerate inconsistent output formatting, plus iterative repair using automated feedback on translation errors, significantly improved translation accuracy.
The authors conclude that reliable production use of LLM code translation would require models with larger windows than tested here.

Terms in this piece · Glossary

LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.

Read the source arxiv.org