Code Transformation Rule Synthesis using LLMs: Potential and Limits
Source
arxiv.org
Author
Axel Allain, Aymeric Blot, Djamel Eddine Khelladi, Mathieu Acher
Date
Context
For repetitive code changes, a LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → can generate a reusable transformation rule that specifies which code to match and how to rewrite it. This study evaluates rules in Comby, GritQL, and Ast-Grep across repair and migration tasks. The authors report that GPT-5.4, from OpenAI, produced transformations closest to reference answers on most benchmarks, while smaller tested models struggled more with complex migrations. They also found a trade-off against a conventional rule-synthesis algorithm: model-generated rules scored better on correctness but worse on applicability. Reusable rules are therefore a promising output for coding assistants, with coverage and correctness requiring separate checks. Study abstract.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
articleAn Empirical Evaluation of Cost-Efficient Large Language Models on Algorithmic Programming TasksChandimal Adikari, Nandika Herath
articleCodeTransBenchmark: Evaluating LLM-based Code Translation and Repair Across Programming LanguagesVera Kowalczuk, Oliver Wei{\ss}l, Severin Kacianka, Andrea Stocco
articleHow effective are traditional test criteria at detecting bugs in large language models generated code?Asma Hamidi, Michael Konstantinou, Renzo Degiovanni, Mike Papadakis