John Yang is a Stanford computer science PhD student who co-created SWE-bench and SWE-agent. His research builds tests and tools for AI systems that work on real software repositories.[1][2][3]
Start with SWE-bench: Can Language Models Resolve Real-World GitHub Issues? →
Education
Links
Website
GitHub 
X LinkedIn Google Scholar Computer science PhD researcher · Stanford University
Background
Yang earned a BS in computer science at UC Berkeley and an MSE in computer science at Princeton, where Karthik Narasimhan advised him. His current Stanford doctoral research is advised by Ludwig Schmidt and Diyi Yang. His personal site identifies language-model agents, evaluation, synthetic data, and software engineering as his research areas.[1]
Contributions
Testing repairs to real software
SWE-bench asks whether a language model can resolve an issue from an existing GitHub repository. Yang co-authored the original benchmark with Carlos E. Jimenez and collaborators. Each task pairs an issue description with a codebase, and the system must produce a patch that passes the relevant tests. The original release covered 2,294 problems across 12 Python repositories. This evaluates a different skill from completing an isolated coding exercise: the system must locate and change code within a larger project.[2]
Interfaces for coding agents
Yang is the first-listed author of SWE-agent, which studies how the tools exposed to a language model affect its performance. Its interface lets an agent inspect repositories, edit files, and run tests. The paper compares interface choices and shows that the surrounding software can change what an agent achieves with a model. For an engineer building a coding assistant, it offers a concrete way to investigate failures in navigation or editing instead of treating every failure as a model limitation.[3]
Building programs from their behavior
ProgramBench extends this research to building a complete program from an executable and its documentation. The agent can experiment with the executable but cannot inspect its source. It chooses the implementation language and architecture, then submits code that is checked against behavioral tests. Yang lists the project among his work. The benchmark probes longer engineering tasks, while its authors explicitly note that behavioral tests cannot cover every possible input.[1][4]
Ideas
In Position: Humans are Missing from AI Coding Agent Research, Yang and co-authors argue that autonomous task completion is an incomplete measure of practical usefulness. They call for evaluations of whether people can communicate intent, verify outputs, steer an agent, and adapt its behavior. This is a proposed research agenda, not evidence that those problems have been solved. It adds a useful question to a benchmark result: can a developer understand and supervise the work behind a passing score?[5]
Tools, writing & talks
Writing, research & talks
Start here
Start here to understand what coding-agent benchmark scores measure: fixing repository issues and checking the resulting patches with tests.
Read the introduction and task examples to understand what SWE-bench measures: whether a model can fix real GitHub issues. This gives you a basis for interpreting coding-agent benchmark claims.
Co-authored paper[2]In a Stanford Benchtalks interview, SWE-bench co-creator John Yang discusses ProgramBench and how each generation of AI coding benchmarks exposed new model failure modes as capabilities advanced.
Catalog workOriginal source- ProgramBenchprogrambench.com
A benchmark that asks agents to recreate programs from executable behavior and documentation, without access to their source.
- Humans are Missing from AI Coding Agent Researchzorazrw.github.io
A position paper on measuring how well people can supervise, verify, and steer coding agents.
Co-authored position paper[5]