Intel / post

Coding Agent Index Adds Safety Refusal Rate Tracking

Source: ArtificialAnlys
Date: Published Sep 18, 2026

Context

Artificial Analysis added safety refusal reporting to its Coding Index, an attempt to explain why models score differently beyond raw capability. A safety refusal happens when a provider or model declines to start or continue a task on safety grounds; when that happens, an may fall back to a different model to keep going, or the attempt is simply blocked. In the latest index version, Anthropic's Claude Fable 5.1 had the highest fallback rate of the models measured, in both Claude Code and Devin Fusion, Cognition's coding agent, with fallback attempts accounting for 8.8% and 7.1% of each agent's index weight respectively, meaning those results partly reflect the performance of whichever model it fell back to rather than Fable 5.1 alone. Artificial Analysis notes that refusal variability, how much has built up in the harness, effort settings, and retry strategies can all affect the observed rates, so the figures describe this specific measurement setup rather than a fixed property of the model.

Terms in this piece · Glossary

AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.
context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.

Read the source x.com