• The comparison used two fuzzing techniques against ten well-known, release-quality human-written Linux utilities as ground truth: classic black-box generational testing and modern coverage-guided (gray-box, mutational) testing with AFL++.
• The failure types diverged by category rather than just count: AI-generated versions were less likely to have memory errors such as buffer overflows, but more likely to have hangs such as infinite loops, compared to the human-maintained code.
• The authors caution that getting reliable code this way required careful practice and human supervision throughout — result quality was highly dependent on the prompts and skill of the person directing the agentic workflow.

• agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.

A concrete, adversarially tested data point against the assumption that AI-generated code is inherently less robust than human-written code, at least for well-scoped utility rewrites.

• articleA Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI AgentsWuyang Dai, Moses Openja, Jiho Shin, Hung Viet Pham, Song Wang
• articleEngineering Reliable Coding Agents: Evaluating and Operating the System Around the ModelStephanie Jarmak
• articleThe reach of a verification tool decides its value: A controlled study of verification surface, artifact quality, and cost in AI coding agentsAchint Mehta

Checking sign-in…

Loading comments…