Can AI Agents Make Open-Ended Scientific Discovery? Evidence from Station
- Source
- arxiv.org
- Author
- Wenyu Du, Stephen Chung
- Date

Context
Can AI agents do scientific work when there is no clear score to chase? A preprint tests this in Station, an environment where several agents simulate a scientific community. The authors add a Supervisor mechanism and periodic Meta Reflection, both meant to keep agents exploring when intermediate metrics are lacking.
To score it, they took the main research question from each of three ICLR oral papers, withheld the results, disabled web access, and counted how many of each paper's original findings (split into criteria) the agents rediscovered. They report 62.7% on average for Station, against 15.4% for Codex Multiagent-v2 and 14.4 to 20.6% for AI Scientist-v2. Ablations indicate the two mechanisms together improve coverage and continuity.
The test covers three papers and the authors' own criteria, and the abstract does not say how overlap with models' data was handled. Two further tasks without reference papers produced some discoveries that matched later researcher reports, which is suggestive, not a general result.
- The authors built tasks from three recent ICLR oral papers, giving agents each paper's research question with results withheld and web access off, then scored how many original findings, split into criteria, were rediscovered.
- Station rediscovered 62.7% of criteria on average, versus 15.4% for Codex Multiagent-v2 and 14.4 to 20.6% for AI Scientist-v2, according to the abstract.
- On two further open-ended tasks with no oracle paper, the authors report that some discoveries closely matched findings researchers reported after the date.
- Training — The process that adjusts a model using examples and a measure of how well it performs.
- multi-agent — Using several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.
- Agent — A system in which a model repeatedly chooses operations, a harness carries them out, and the results return to the model to inform its next choice.
- knowledge cutoff — The date after which a model saw no training data — everything later has to reach it through search, tools, or the context window.
articleDiscoveryAgent: A Framework for Scientist-centric Agentic AIOwen Queen, Ioana Ciuca, Emma Dann, Jean Somalwar, Carine You, Steven Dillmann, Viraj Manwadkar, Evan Gorrell, Harrison G Zhang, Jonathan K Pritchard, Risa H Wechsler, James Zou
articleAutomating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?Ming Zhong, Stacy Li, Nicholas Carlini, Matthew Jagielski
articleCan AI Agents Conduct Open-Ended AI Research? Early Evidence from Two Case StudiesPeter Kirgis, Sayash Kapoor, Andrew Schwartz, Stephan Rabanser, David Africa, Konstantinos Voudouris, Viet Nguyen, Toby Pilditch, Magda Dubois, Harry Coppock, Cozmin Ududec, Nitya Nadgir, Matilda Orona, Tilman Bayer, Derrick Chan-Sew, Yue Ling, Abhishek Shetty, Helen Toner, Gillian Hadfield, Seth Lazar, Steve Newman, Shoshannah Tekofsky, Rishi Bommasani, Arvind Narayanan
Checking sign-in…
Loading comments…