SynthCorp: A Platform for Creating Benchmarks for Data Agents
Source
arxiv.org
Author
Jiani Zhang, Manav Garg, Eugenie Lai, Cheng Zhen, Eva Sharma, Yang Pan, Yaming Lin, Alon Halevy
Date
Why it matters
Data-AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → benchmarks saturate quickly. A simulation-driven generator with verifiable gold programs lets you build harder, targeted evaluations of agents across databases, documents and catalogs.
Key takeaways · AI-distilled
SynthCorp works in four steps: synthesize reusable event types compiled to simulation primitives, schedule routine and disruption events on a virtual calendar, execute it while logging every operation, then derive tasks and gold programs from that log.
The initial benchmarks cover seven data estates with up to 1K operational tables and 256 tasks across five domains; the authors report a near-zero error rate on deterministic checks such as foreign-key violations and undeclared writes.
Gold programs are verified by auditing candidate queries against causal provenance in the simulation log, with no manual annotation, including multi-event composite tasks that expose reasoning failures in state-of-the-art agents.
Evaluating three models across four data-context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition → settings, the authors find the generated benchmarks separate both data contexts and model tiers.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
articleSlackBench: Benchmarking Agents on Collaborative Projects Grounded in Real Code RepositoriesMingxuan Li, Huanzhi Mao, Ruitao Zheng, Jarod Stanbury James, Yunze Li, Natanya Anderson, Joseph E. Gonzalez, Alex Dimakis
articleBenchmarking Hybrid Deep Research Across Database Querying and Web SearchRuofan Wu, Peiran Xu, Xiaolong Li, Fan Shu, Soyoung Yoon, Yite Wang, Xiaodong Yu, Boyi Liu, Feng Yan, Debiao Li, Yuxiong He, Zhewei Yao
articleAgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSLDipin Khati, Shubham Mehrotra, Bin Bi, Zhou Yu, Denys Poshyvanyk, James Zhu, Sitaram Asur, Phil L. Mui