
Frontier models fully resolve under 1% of from-scratch ProgramBench tasks. Converting open-source command-line programs into source-free environments — only a compiled reference executable plus docs — yields trajectories that take a 27B model from 38% to 49.5%.
articleSpecFirst: Behavioral Specification Elicitation as a First-Class Step in Agent-Based Program Synthesis from ScratchYihao Chen, Shi Chang, Feng Lin, Khaled Chawa, Boyuan Chen, Shaowei Wang, Ahmed E. Hassan
videoBenchmarking Coding Agents on New vs Legacy Codebases — Denys Linkov, WisedocsAI Engineer
articleLoopsBench: From Harness Engineering to Loop Engineering in Benchmarking Coding AgentHan Li, Zhemin Fang, Rili Feng, Yingqi Zhao, Jiaheng Liu, Pengfei Gao, He Ye, Dayi Lin, Qingwei Lin, Saravan Rajmohan, Dongmei ZhangChecking sign-in…
Loading comments…