AutomationBench
zapier.com- Category
- Developer Tools
- Rank
- No. 1192Tools index
Previous survey · No. 1167 ·
- Listed in
- Find AI benchmarks
- Type
- TOOL
- Use case
- Model & Agent Evaluation
- GitHub
- 353 stars
- Date
About
Tests agents on business workflows across simulated applications, including sales, support, finance, and HR, grading the resulting system state.
What it does
AutomationBench is Zapier's benchmark for AI agents doing office work. Each task drops the agent into a fake company: a seeded CRM, inbox, calendar, spreadsheets and dozens of other mock business apps, plus a triggering message such as an incoming email. The agent works through tool calls with no human in the loop. When it stops, a checker compares what the mock apps now contain against a list of expected facts, so a task passes only if the records actually changed the right way, not if the agent merely said the right thing.
Why it's ranked here
It measures the outcome that matters for workflow agents, the end state of the systems, and it is built so that measurement is hard to game. Six business areas carry 100 public tasks each, every task is checked by code rather than a judge model, and the scorer refuses credit for checks that were already true before the agent acted. It is MIT licensed, runs from one command or a hosted environment, and publishes reference pass rates for current frontier models that top out near half, so the benchmark still separates them.
What's good
The anti-gaming rule in the scorer is concrete: a check that already passes at the start earns nothing, and breaking it costs a failure. Each run also fingerprints the exact task inputs it used, so results from a changed task cannot be silently mixed with old ones. Agents can only reach the services a task actually sets up; a write to an unrelated vendor fails instead of vanishing into untracked state. You get two scores per task, a strict pass and a partial fraction usable as a training reward, plus a local results viewer that compares up to five runs side by side.
Tradeoffs
The official leaderboard runs on a private, harder task set that is never released, and the README says local scores may not match it and that private tasks get harder across versions. So public numbers are directional, not a reproduction. The apps are simulations derived from public API docs, which tests tool use against modelled behavior rather than real vendor quirks. It needs a recent Python, 3.13 or newer. The maintainers say plainly that buggy or unfair tasks remain, and the changelog shows fairness fixes landing release after release, so scores can shift between versions.
How to use it well
Use it to compare models or agent setups on multi-step business automation before you trust one with real accounts. Pin the release you test against, since tasks change between versions. Start with a single area and a handful of tasks to check cost, then run the full set; the warm-up tier of 200 easy tasks is a quick sanity check that tool calling works at all. Try the three tool styles, since searching for tools differs from having them handed over. It does not cover coding, browsing or chat quality, and it will not tell you how an agent behaves against live vendor APIs.
Technical notes+
automationbench/runner.py subclasses verifiers' StatefulToolEnv as AutomationBenchEnv; setup_state builds a pydantic WorldState from info.initial_state (after strip_none_values undoes HuggingFace Dataset null-padding), computes allowed_services from seeded keys, assertion types and zapier_tools via compute_allowed_services, and filters tool_defs per task unless toolset is api or meta-tools mode. The zapier toolset registers only two meta-tools (search_tools, execute_tool), with --search-top-k capping discovery. automationbench/task_contract.py hashes example_id, prompt, a canonicalised assertion multiset (order-insensitive only for keys like exact_to and body_contains) and initial_state into a sha256 under schema automationbench.task-contract.v1. automationbench/rubric/__init__.py implements partial_credit with free-assertion exclusion, an excluded=False override for do-nothing inverse tasks, and weights task_completed_correctly at 0.0 in the vf.Rubric. pyproject.toml requires Python >=3.13 and exposes the auto-bench script; the datasets package is imported but not declared directly.
Observed
- License
- MIT for original code and data; derived third-party API schema representations are not claimed (LICENSE)
- Language
- Python, requires 3.13 or newer (pyproject.toml)
- Install and run
- uv sync, then the auto-bench CLI; also packaged as a Prime Intellect hosted environment
- Evaluation framework
- Built on the verifiers library StatefulToolEnv
- Public task set
- 600 scored tasks, 100 in each of six business domains, plus 200 unscored simple tasks
- Grading
- Programmatic assertions on final simulated state; no judge model
- Official scores
- Computed on a held-out private task set not included in the repository
- Tool styles
- api, zapier (search and execute meta-tools), limited_zapier
- Model providers
- OpenAI-compatible endpoints by default; Anthropic models detected by name prefix
Read from README.md, pyproject.toml, LICENSE, automationbench/__init__.py, automationbench/runner.py, automationbench/task_contract.py, automationbench/rubric/__init__.py, automationbench/tool_wrapper.py, automationbench/domains/__init__.py, CHANGELOG.md.
What it can do
Test agents on simulated business workflows (sales, support, finance, HR)
Agent actions → Workflow execution results
Grade resulting system state after agent task completion
System state → Grading score
Intel on AutomationBench
Tags
Tech Stack
Media
Comments (0)
No comments yet
Editorially curated, with community endorsements as a secondary signal. Corrections welcome.