From SWE-bench to ProgramBench: The Next Generation of Coding Evals

Stanford SAIL's post points to a conversation between John Yang and Vincent Chen about ProgramBench. Where SWE-bench, also created by Yang, scopes a task to fixing one GitHub issue within an existing codebase, ProgramBench asks a model to produce an entire working program, with the solution space and implementation approach left entirely up to the model, evaluated by whether the resulting artifact actually runs correctly.

As Yang has described the benchmark elsewhere, every frontier model scored 0% on the 200 launch tasks, which include reconstructing programs like FFmpeg, SQLite, the PHP interpreter, and the tinycc compiler. About a week after launch, GPT-5.5 became the first model to pass a single task, cmatrix, a terminal animation program, notably choosing to implement it in Python rather than the reference language.