Partial workflow success and complete task delivery are different measures. In its Intelligence Index update, Artificial Analysis explains that AutomationBench-AA gives credit for objectives completed, unless a guardrail violation makes the task score zero. A separate measure counts only workflows where every objective is met without a violation.
The distinction prevents reading a partial-credit score as the proportion of jobs an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → can finish unaided. The update also replaces older tests with harder terminal tasks and broader business workflows. Its model scores describe that revised benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition → mix; they should not be treated as directly interchangeable with numbers from an earlier index version.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
