PAWS gives financial-policy simulations a historical record against which to check their agents' behavior. The dataset connects 36 US policy episodes to dated news, actions attributed to organizations, and daily market context windowThe maximum amount of text a model can consider at once — its working memory for the current conversation or task.Full definition →. Each extracted action retains a link to its supporting news, allowing researchers to inspect the evidence behind a simulated response.
Its replay study also identifies a scoring problem: high overall accuracy can coexist with missed rare actions. Evaluating a policy simulation therefore requires checking which responses it reproduces and when they occur. The dataset supplies evidence for that comparison; its alignmentThe work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.Full definition → of actions and prices does not itself establish causation.
Key takeaways · AI-distilled
PAWS covers 36 verified U.S. financial and economic policy episodes, 12,727 policy-linked news records and 65,291 stakeholder actions, each linked to the news that supports it.
Each action carries a multi-layer event frame (interaction mode, financial-action family and subtype, semantic attributes), with entities resolved to normalized organizations and actions aligned to daily market returns for replay.
Case studies of the 2008 short-selling ban and 2001 decimalization recovered documented policy timelines and associated market patterns in both dense and sparse news settings.
A replay study found that high accuracy can mask failure to detect rare stakeholder actions, pointing to action timing and calibrationHow well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.Full definition → as the central challenges.
Terms in this piece · Glossary
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
alignment — The work of making AI systems actually pursue what their builders and users intend, rather than something subtly or dangerously different.
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
calibration — How well a model's confidence matches reality — a calibrated model saying "90% sure" is right about 90% of the time.
articleWorld of SWECraft: Agent Collaboration at Scale via Simulated Software WorldsShannon Zejiang Shen, Yapei Chang, Qiyao Liang, Benjamin Newman, Jenny Liang, Rulin Shao, Ziqiao Ma, Junjie Oscar Yin, Yung-Sung Chuang, Weijia Shi, Yejin Choi, Luke Zettlemoyer, Pang Wei Koh, Mohit Iyyer, Tongshuang Wu, David Sontag
articleOpenAgentFlow: Enabling System-Wide Safety Boundaries for Heterogeneous AI Agent FleetsDongsheng Chen, Xiangyu Zhao, Xin Yao, Xuetao Wei
articleA Large-Scale Empirical Study of Quality Assurance Practices and Gaps in AI AgentsWuyang Dai, Moses Openja, Jiho Shin, Hung Viet Pham, Song Wang