FAB (Finance Agents Benchmark)

At a glance

Finance Agents Benchmark tests whether model agents can answer due-diligence questions from a synthetic company data room. It is worth using as a repeatable evaluation fixture if you want to inspect the evidence an agent finds and see which individual rubric checks it misses.

Good fit

Model evaluators comparing document research across the same tasks and grading setup.

Consider something else when

Skip it if you need evidence of reliability on your own deal documents or financial advice.

Start here

Read one task and its rubric, then inspect a published trace before running anything. Check which document supports each answer and compare the full-task result with the criterion-level results.

Cost and requirements

MIT code and CC BY 4.0 data. Reproduction requires Python 3.11 or later, uv, Docker or Podman, and model credentials. Agent and judge calls can incur provider charges.

Limits to know

The published run covers one synthetic company and uses a model judge. A task passes only when every criterion passes. The tool sandbox is documented as network-disabled, but the harness still calls hosted models.

Related work

• OpenAI, provider used for the published model judge (context)