Vibeleaderboard
← All Intel
Intel / article

Can AI Agents Make Open-Ended Scientific Discovery? Evidence from Station

Source
arxiv.org
Author
Wenyu Du, Stephen Chung
Date

Context

Can AI agents do scientific work when there is no clear score to chase? A preprint tests this in Station, an environment where several agents simulate a scientific community. The authors add a Supervisor mechanism and periodic Meta Reflection, both meant to keep agents exploring when intermediate metrics are lacking.

To score it, they took the main research question from each of three ICLR oral papers, withheld the results, disabled web access, and counted how many of each paper's original findings (split into criteria) the agents rediscovered. They report 62.7% on average for Station, against 15.4% for Codex Multiagent-v2 and 14.4 to 20.6% for AI Scientist-v2. Ablations indicate the two mechanisms together improve coverage and continuity.

The test covers three papers and the authors' own criteria, and the abstract does not say how overlap with models' data was handled. Two further tasks without reference papers produced some discoveries that matched later researcher reports, which is suggestive, not a general result.

Key takeaways · AI-distilled
  • The authors built tasks from three recent ICLR oral papers, giving agents each paper's research question with results withheld and web access off, then scored how many original findings, split into criteria, were rediscovered.
  • Station rediscovered 62.7% of criteria on average, versus 15.4% for Codex Multiagent-v2 and 14.4 to 20.6% for AI Scientist-v2, according to the abstract.
  • On two further open-ended tasks with no oracle paper, the authors report that some discoveries closely matched findings researchers reported after the date.
Terms in this piece · Glossary
  • Training — The process that adjusts a model using examples and a measure of how well it performs.
  • multi-agent — Using several AI agents on one problem — splitting work in parallel, checking each other, or filling different roles like planner and reviewer.
  • Agent — A system in which a model repeatedly chooses operations, a harness carries them out, and the results return to the model to inform its next choice.
  • knowledge cutoff — The date after which a model saw no training data — everything later has to reach it through search, tools, or the context window.
Recommended reads
Comments

Checking sign-in…

Loading comments…