Vibeleaderboard
← All Intel
Intel / article

Beyond Source Code: Evaluating Natural Language Specifications for LLM-Based C-to-Rust Translation

Source
arxiv.org
Author
Biruk Tadesse, Lawrence Arkoh, Marcelo D'Amorim, Wesley K. G. Assun\c{c}\~ao
Date
From the Daily Brief

Researchers compared three ways of asking GPT-5.2 and GPT-4o-mini to translate 300 C programs from IBM's CodeNet dataset into Rust: from the source code alone, from a natural-language specification alone, and from both. Code-only translation achieved the highest test pass rate, 98.6%, but often mirrored C's implementation patterns. Specification-only translation produced the most idiomatic and compact Rust, with roughly 6x fewer unsafe blocks and 2.4x fewer Clippy style warnings, but without validation its pass rate fell as low as 86% for the best model. An iterative step that validates and refines the specification against tests raised that to 95.3%. Giving the model both inputs reached 98.3% and, in the authors' assessment, the best balance between correctness and maintainability. For teams running LLM-assisted migrations, the study puts numbers on a choice that is usually made by habit: the input format, not only the model, shapes how much unsafe code survives translation.

Read the 2026-11-16 Brief →

Context

Asking a to translate C into Rust from a plain-English description of what a program does, instead of from the C code itself, produced cleaner and safer-looking Rust in this study, at some cost in correctness and a large cost in compute. Rust is a programming language that enforces memory safety when code is compiled, and moving old C code to it appeals because memory errors in C are a long-standing source of security bugs. Existing automatic translators tend to produce what the authors call C written in Rust syntax, including unsafe blocks, sections where Rust's safety checks are switched off. In Beyond Source Code, Biruk Tadesse and colleagues at North Carolina State University test whether seeing the C code anchors a model to C habits. They translated 300 short C programs from IBM's CodeNet collection of competitive-programming solutions with OpenAI's GPT-5.2 and GPT-4o-mini in three ways: from the code alone, from a specification alone, and from both. GPT-4o-mini translated too few programs correctly to carry the study, so the main results use GPT-5.2.

The specification is an LLM-written description of behavior in nine fixed sections, among them inputs, outputs, preconditions and error handling, and its prompt forbids naming variables, library calls or loop structure. Unchecked specifications left out details the program's output depended on. In one example, the specification for a cube-root program omitted the starting guess for its Newton's-method loop, so the Rust picked a different one and computed a different answer. The authors therefore added a check: a model rewrites the C program from the specification alone, runs it against the program's tests, and if it fails, revises the specification from a written diagnosis of the gap, up to three times. Every translation also got up to three repair attempts using compiler and test feedback. With GPT-5.2 writing both the specifications and the Rust, specification-only translation passed the tests for 259 of 300 programs (86.3%) with unchecked specifications and 286 (95.3%) with checked ones, against 98.6% for code only and 98.3% for code plus checked specification.

The quality differences run the way the authors expected. Code-only translations contained 82 unsafe blocks covering 342 lines; specification-only translations contained 14 blocks covering 14 lines, with code plus specification between them at 22 blocks. Clippy, Rust's standard linter, raised 1,084 style warnings on the code-only output against 456 for specification-only. In a manual review of the 285 programs that all three settings translated correctly, code-only output kept the C program's variable names in 98% of cases and its C-style loops in 91%, while specification-only translations swapped in a simpler standard approach, such as a library sort in place of a hand-written bubble sort, in 90.7% of cases against 1.7% for code only. On a maintainability index, code-only output scored 30.94, against 37.89 for specification-only and 38.96 for code plus specification. The check is expensive: by the authors' -count estimate, code-only translation cost $4.34 to $4.61 across the pipelines, while the two checked pipelines came to $17.29 and $17.18, with $13.01 of that spent on validation.

The evidence supports a narrower claim than a better way to migrate real C systems. The programs are short and self-contained, the specification format leaves out memory ownership, shared state and cross-function contracts, which the authors note are central to repository-scale migration, and the study covers two OpenAI models and one language pair. The specifications were checked against the same tests later used to grade the Rust, and the authors warn that this may tune them to those tests and inflate pass rates in the specification-based settings. Revision also pulled some implementation detail back into the specifications, and the authors did not measure how often. The maintainability and linter figures are structural proxies, not judgments by Rust developers. A Ship-of-Theseus translation of the 12,500-line iodine tool takes the opposite route on a real codebase, producing a faithful but unidiomatic Rust baseline first and then rewriting it into idiomatic Rust step by step.

Key takeaways · AI-distilled
  • On 300 IBM CodeNet C programs translated by GPT-5.2 and GPT-4o-mini, code-only translation passed the most tests (98.6%) but tended to carry C implementation patterns into the Rust, per the abstract.
  • Translating from a natural-language specification alone produced the most idiomatic, compact Rust, with about 6x fewer unsafe blocks and 2.4x fewer Clippy style warnings than code-only translation.
  • Specifications without validation cut test pass rates to as low as 86% for the best model. Adding iterative test-based validation and refinement of the specification raised that to 95.3%.
  • Giving the model both code and specification offered the best balance between correctness and maintainability in the study, with a 98.3% test pass rate against code-only's 98.6%.
Terms in this piece · Glossary
  • LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Recommended reads
Comments

Checking sign-in…

Loading comments…