Only one model clearly clears 50% on irreversible, policy-constrained enterprise workflows. Expect agents to fail often on real operational tasks, and use state-based SQL grading like this when testing your own.
Key takeaways · AI-distilled
EnterpriseOps-Gym-AA puts agents in live systems across 8 domains through MCPThe Model Context Protocol — an open standard that lets any AI assistant plug into any tool or data source without custom integration code.Full definition → servers exposing 164 database tables and 512 tools, where many actions are irreversible and cannot be brute-forced.
Each task is graded by executable SQL checks on the final database state for goal completion, state integrity, policy compliance and unintended side effects, so one missed constraint usually fails the task.
Claude Fable 5 led at 51% of tasks despite passing 79% of individual checks. HR (26%) and IT Service Management (28%) were hardest, while Email (58%) and Drive (53%) were markedly easier.
Cost ranged about 90x without reliably buying score: DeepSeek V4 Pro reached 40% at about $0.01 per task, and GLM-5.2 scored 43% for $0.10 versus Opus 4.8's 44% at $0.79.
Terms in this piece · Glossary
MCP — The Model Context Protocol — an open standard that lets any AI assistant plug into any tool or data source without custom integration code.
