Intel / article

SpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from Scratch

Source: arxiv.org
Author: Yihao Chen, Shi Chang, Feng Lin, Khaled Chawa, Boyuan Chen, Shaowei Wang, Ahmed E. Hassan
Date: Published Jul 30, 2026

Context

Recreating a program from documentation and an executable reference requires discovering its behavior before implementing it. SpecFirst makes that discovery a separate phase: one probes the reference program and combines the results with documentation into a structured specification; another uses that specification to write code. Across 200 ProgramBench tasks and four models, the authors report higher test-pass rates and broader exploration than a workflow that mixes investigation and implementation. The specification gives the coding stage a persistent account of observed requirements, helping prevent early misunderstandings from carrying through the build. Research abstract.

Key takeaways · AI-distilled

Building a program from scratch is where agents collapse. Given only documentation and an execute-only binary to probe for behavior, frontier models solve under 1% of ProgramBench instances, despite doing well when an existing codebase supplies .
The failure mode is the single pass that mixes reading docs, probing the binary, and writing code. Agents under-explore, lose the behavioral intent as context drifts, and bake early misreadings into the final implementation.
SpecFirst splits the job in two. A spec agent probes the binary and merges what it observes with the documentation into a structured written specification; a separate synthesis agent then implements against that fixed reference.
Across all 200 ProgramBench instances and four models, the split lifted test pass rates by 6.9 to 21.3 percent and binary exploration coverage by 9.4 to 18.5 percent, and agents began writing real code earlier and sustained it longer.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.

Key quotes

“LLM-based agents excel at software engineering tasks where an existing codebase provides context, but constructing a program from scratch remains fundamentally harder.”

“Recent benchmarks such as ProgramBench quantify this gap: given only natural-language documentation and an execute-only binary as a behavioral oracle, even frontier models solve fewer than 1% of instances.”

“SpecFirst consistently outperforms the single-loop baseline, improving test pass rates by 6.9%-21.3% and binary exploration coverage by 9.4%-18.5%, all statistically significant.”

“Our results demonstrate that an explicit requirements-engineering phase is an effective paradigm for from-scratch program construction.”

Read the source arxiv.org