DevIntent: How Much Does LLM-Generated Code Violate Developer Intent?
- Source
- arxiv.org
- Author
- Susana Haing, Natan Vidra, Spurthi Setty
- Date
Context
DevIntent tests whether generated code can pass stated checks while missing a requirement left out of the prompt. Its 49-problem pilot removes constraints from fuller specifications and checks them separately. One example requests unique list values while withholding the requirement that they also be sorted. The paper explains how both interpretations can pass the visible check.
For Anthropic's Claude Sonnet 4.6 and OpenAI's GPT-4.1, the authors report mean intent-violation rates of 54.5% and 63.5%. These rates count hidden-constraint failures among solutions that pass stated tests, averaged across qualifying problems. They measure sensitivity to deliberately underspecified tasks, rather than the failure rate of ordinary coding requests.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
articleHow well LLM-based test generation techniques perform with newer LLM versions?Michael Konstantinou, Renzo Degiovanni, Mike Papadakis- postLLMs still produce bugs, but those bugs are different than what they used to be.Boris Cherny
articleOn the Robustness of LLMs' Internal Representation of Code CorrectnessFrancisco Ribeiro, Sohaila Abdulsattar, Renata Gonzalez, Mahmoud Kassem, Sarah Nadi
Checking sign-in…
Loading comments…