From Cosmetic to Structural: A Paired Experiment on Metric-Guided Agentic Refactoring
Source
arxiv.org
Author
Nadim Hagatulah, Markus Borg, Emma S\"oderberg
Date
Why it matters
Giving a coding agent code-health feedback through MCP shifts it from cosmetic to structural refactors, but below roughly CodeHealth 5 to 7 it can cause regressions. That tells you when guided refactoring is safe to automate.
Key takeaways · AI-distilled
Per the abstract, giving the gptme coding AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → CodeScene code-smell reports and CodeHealth scores through an MCPThe Model Context Protocol — an open standard that lets any AI assistant plug into any tool or data source without custom integration code.Full definition → server raised the rate of effective refactorings by 21 to 26 percentage points versus unguided runs.
The study ran a paired experiment on 24,620 single-file competitive programming solutions in C++, Java, and Python, comparing guided and unguided refactorings of the same code.
The authors report that guidance pays off only above language-specific break-even points of CodeHealth 4.9 to 7.1; below them, the structural edits it encourages increasingly risk regressions.
Guided runs cost more: about 3 to 5 times the tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → over roughly twice as many turns, and they shifted failures toward logical test failures and introduced new coupling smells.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
MCP — The Model Context Protocol — an open standard that lets any AI assistant plug into any tool or data source without custom integration code.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
