Coding Agent Index v1.5 Reports Safety Refusal Fallbacks

Artificial Analysis now reports safety refusals in its Coding Agent Index, and the first figures show that some models' scores partly reflect another model's work. According to its announcement of index version 1.5, a safety refusal occurs when a provider or model declines to start or continue a task on safety grounds. The coding agent then either falls back to another model to keep going or stops the attempt with a block.

Anthropic's Claude Fable 5.1 had the highest fallback rates in both agents Artificial Analysis names, Claude Code and Devin Fusion. Its attempts that fell back to another model account for 8.8% of the index's weight in Claude Code and 7.1% in Devin Fusion, and Artificial Analysis says those results therefore include the fallback models' performance. Fable 5.1's score in each agent is a blend of its own work and the work of whichever model took over, and because the announcement's text does not name the fallback models, it does not show whether that share raised or lowered the score.

The rates also depend on the setup rather than the model alone. Artificial Analysis lists refusal variability, the context that builds up in the agent harness during a task, effort settings and retry strategies as factors that can change the observed rates, so the same model could refuse more or less often in a different configuration. These figures belong to version 1.5; Artificial Analysis later released version 2.0 of the index.