Can AI Agents Conduct Open-Ended AI Research? Early Evidence from Two Case Studies
Source
Peter Kirgis, Sayash Kapoor, Andrew Schwartz, Stephan Rabanser, David Africa, Konstantinos Voudouris, Viet Nguyen, Toby Pilditch, Magda Dubois, Harry Coppock, Cozmin Ududec, Nitya Nadgir, Matilda Orona, Tilman Bayer, Derrick Chan-Sew, Yue Ling, Abhishek Shetty, Helen Toner, Gillian Hadfield, Seth Lazar, Steve Newman, Shoshannah Tekofsky, Rishi Bommasani, Arvind Narayanan
Author
Peter Kirgis, Sayash Kapoor, Andrew Schwartz, Stephan Rabanser, David Africa, Konstantinos Voudouris, Viet Nguyen, Toby Pilditch, Magda Dubois, Harry Coppock, Cozmin Ududec, Nitya Nadgir, Matilda Orona, Tilman Bayer, Derrick Chan-Sew, Yue Ling, Abhishek Shetty, Helen Toner, Gillian Hadfield, Seth Lazar, Steve Newman, Shoshannah Tekofsky, Rishi Bommasani, Arvind Narayanan
Date
Key takeaways · AI-distilled
Existing AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition →-research evalA repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.Full definition → fail two ways: narrow verifiable tasks exclude open-ended work by construction, and submitting AI-written papers to blind peer review inherits a process that is overstretched, stochastic, and noisy.
The split result is the finding: agents finished all the engineering with no human help, yet made no substantial progress on the research question itself. Capability is unevenly distributed across the research lifecycle.
The five failure modes were misjudging the bar for publishable work, uncreative responses to flaws in the research design, ineffective backtracking out of dead ends, poor resource awareness, and instruction drift.
Grading comes from the paper's own authors rather than outside reviewers, since they already know how hard the question is and what a real answer looks like. That is what makes the rejection of both submissions meaningful.
Terms in this piece · Glossary
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
Why it matters
If you are building or evaluating research/deep-work agents, this gives you a realistic evaluation design (have agents attack an unpublished paper's core question, then let the authors grade it) plus a concrete map of where agents fail — publication-standard judgment and escaping dead ends — rather than the narrow, saturating benchmarks most agent evals rely on.
Key quotes
“The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions.”
“We identify five recurring failure modes: poor judgment about the bar for publishable research, uncreative responses to shortcomings in the research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift.”
“Our results provide early evidence that today's agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle.”
