Vibeleaderboard
← All Intel
Intel / article

DiscoveryAgent: A Framework for Scientist-centric Agentic AI

Source
arxiv.org
Author
Owen Queen, Ioana Ciuca, Emma Dann, Jean Somalwar, Carine You, Steven Dillmann, Viraj Manwadkar, Evan Gorrell, Harrison G Zhang, Jonathan K Pritchard, Risa H Wechsler, James Zou
Date

Context

End-to-end AI scientist systems such as Kosmos take a dataset and a goal and hand back a finished report, which leaves the scientist either to trust it or to rebuild it. DiscoveryAgent, from researchers at Stanford University and partner institutions including SLAC and UC Berkeley, splits the work differently. Agents carry out the analyses for one hypothesis at a time, the scientist chooses or rewrites the next hypothesis between rounds, and every step is saved so that separate audit agents can answer questions about how a result was produced. The paper's most useful evidence concerns that audit layer, and it comes from small studies.

Inside each round, a director sends tasks to sub-agents for coding and data analysis, literature and web search, and domain data access, while a review agent periodically critiques the process. Between rounds, a written memory report accumulates findings, including negative ones, and the research code sits under version control. Every query, report and code snapshot is checkpointed. An audit agent attached to that record can explain why an analysis was run, point to the code behind a claim, reproduce a result, or let the scientist roll back to an earlier state.

Two tests measure whether that record helps. In the first, the team took 50 claims from the final reports of DiscoveryAgent and two other agents on a variable-star task and altered 25 of them to inflate results. A GPT-5.6-Sol auditor reading DiscoveryAgent's checkpoints flagged the altered claims with an F1 score of 0.965, higher than auditors reading the other agents' notebooks and logs; given only the reports, its score fell to 0.268. In the second, reviewers had one hour to find errors deliberately planted in a paper written from an earlier DiscoveryAgent run. With the audit agent, reviewers recovered an average of 3.50 planted errors, against 0.25 for reviewers who had only the paper and code. That ratio is the abstract's 14x figure, and it rests on four reviewers per group. The audit layer also caught a real failure: the agent had downloaded held-out ProteinGym test data despite instructions, and the team restored a checkpoint from before that round, tightened the and reran.

The modeling results are competitive rather than decisive. On variable-star classification, DiscoveryAgent's model reached 86.23% balanced accuracy, ahead of an expert-feature baseline at 83.56% but not significantly ahead of the Kosmos agent at 85.12%. On protein variant-effect prediction, the scientist-steered run ranked second of 113 methods behind one supervised predictor, and beat the autonomous run on 134 of 217 assays, though the authors say differences in models and budgets keep them from crediting steering alone. The three expert case studies, in astronomy, insurance economics and immunology genetics, are single demonstrations. The authors also note the system is expensive, handles only computational work, and lacked component ablations because of cost. What the paper does show is that how an agent's research history is organized affects whether its mistakes can be found, and the test-data leak shows why that matters.

Key takeaways · AI-distilled
  • Scientists working with DiscoveryAgent's audit agents found 14 times more valid errors than scientists doing manual review, and the audit agents alone caught errors at higher rates than earlier AI scientist agents, the authors report.
  • The audit design rests on checkpointing: each research round leaves records that audit agents query to trace a claim back, reproduce the analysis and inform the next round's decisions.
  • Between research rounds, scientists select, refine or replace the hypotheses the system pursues, which keeps research direction with them while the investigation itself runs autonomously.
  • Expert-led case studies produced a photometric-redshift estimator with 0.971 AUROC on low-redshift classification and documented an 83% rise in Florida homeowners-insurance premiums from 2013 to 2023, with a growing, regressive burden.
Terms in this piece · Glossary
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
Recommended reads
Comments

Checking sign-in…

Loading comments…