Vibeleaderboard
← All Intel
Intel / article

An AI-Powered Multiverse Analysis of Empirical Finance Research: Evidence from Reused Natural Experiments

Source
arxiv.org
Author
Ruijiang Gao, Steven Chong Xiao
Date
Why it matters

Agents rebuilt data and re-ran tens of thousands of specifications across hundreds of papers, showing agentic reproduction at a scale humans rarely attempt and what it reveals about published results.

Key takeaways · AI-distilled
  • AI agents rebuilt the data for 324 empirical finance papers that reuse 13 natural experiments, reproduced each paper's main estimate, and re-estimated it across grids totaling 62,409 defensible specifications.
  • Only 42% of the main estimates for signed-effect claims reproduced under the papers' own specifications. Across all defensible specifications, on average 25% kept the published sign with p < 0.10.
  • Regression-discontinuity claims were more robust than difference-in-differences ones. Within an experiment, first-order effects, larger published t-statistics, findable data or code, top journals and fewer earlier papers on the experiment all went with more robust claims.
  • Claims on experiments that many papers reuse were not less robust on average, the authors report, even though multiple testing across reused experiments was the concern that motivated the study.
Recommended reads
Comments

Checking sign-in…

Loading comments…