Every tested model violates business rules during workflow automation, so guardrail checks cannot be skipped. Cheaper models can match pricier ones on cost per completed task.
Key takeaways · AI-distilled
AutomationBench-AA runs 657 Zapier-derived tasks across 40 simulated SaaS apps, where agents discover REST endpoints themselves and are graded against nearly 12,000 assertions on the final system state, with a 50-turn cap.
Assertions are either objectives to achieve or guardrailsThe checks around a model that block bad inputs and outputs — filters, validators, and permission rules the model itself can't override.Full definition → that already pass and must not be broken; the headline score counts objectives completed without violating any guardrail, unlike Zapier's hosted leaderboard.
Every model broke business rules, from 0.46 violations per task (Gemini 3.5 Flash) to 1.26 (Qwen3.7 Plus). Gemini 3.5 Flash scored 42.6% at $0.49 per task, matching GPT-5.5 (xhigh) at 42.1% for about 37% of the cost.
Claude Fable 5 led at 48.6% but fell back to Opus 4.8 on about 18% of tasks, which AA says likely limited its uplift over Opus 4.8 at 48.5%. Finance objectives were completed at about a third, roughly half the rate of Support and Operations.
Terms in this piece · Glossary
guardrails — The checks around a model that block bad inputs and outputs — filters, validators, and permission rules the model itself can't override.
