OpenDiscoveryTrace: Process Traces for Evaluating AI Scientist Workflows

OpenDiscoveryTrace gives researchers records of the steps AI scientific agents took, so they can investigate how an answer was produced. The dataset records tool calls, observations, errors and revisions across tasks in fields including drug discovery and materials science.

That detail can expose differences hidden by a final success score. The authors' pilot analysis reports similar completion rates alongside substantially different error patterns. A team evaluating research agents could use the recorded traces to examine whether failures arose during tool use or reasoning, and whether subsequent actions corrected them. The pilot's judgments were made by language models.

“Existing benchmarks for autonomous AI scientists evaluate only final outputs---generated code, hypotheses, or papers---yet discard the reasoning process by which those outputs were obtained.”