Intel / article

A Study of the Reliability of Agentic AI-Generated Programs

Source: arxiv.org
Author: Ayesha Shafique, Barton P. MIller, Elisa R. Heymann
Date: Published Sep 17, 2026

Context

Researchers tested whether code written by an autonomous coding is actually as reliable as human-written code once both are put through real (arxiv.org). They had an agent rebuild ten well-known, release-quality Linux utility programs so the output could be compared against a concrete human-written ground truth, then fuzzed both versions with classic random testing and with AFL++'s coverage-guided testing. The AI-generated utilities were typically as reliable as, and in many cases more reliable than, the latest human-written versions, though the failure patterns differed: the AI code was less likely to have memory errors like buffer overflows but more likely to hang in infinite loops. The authors are explicit that this result depended on careful practice and human supervision during generation, since the quality of the output tracked the prompts and of the person directing the agentic workflow, not the agent alone.

Key takeaways · AI-distilled

The comparison used two fuzzing techniques against ten well-known, release-quality human-written Linux utilities as ground truth: classic black-box generational testing and modern coverage-guided (gray-box, mutational) testing with AFL++.
The failure types diverged by category rather than just count: AI-generated versions were less likely to have memory errors such as buffer overflows, but more likely to have hangs such as infinite loops, compared to the human-maintained code.
The authors caution that getting reliable code this way required careful practice and human supervision throughout — result quality was highly dependent on the prompts and skill of the person directing the agentic workflow.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
red teaming — Deliberately attacking your own AI system to find what makes it fail before someone else does.
agent skill — A reusable instruction file that teaches an agent how to do one job well — the procedure, the tools, and what counts as done.

Read the source arxiv.org