Auto mode is now the default in Claude Code for Pro, Max, and Team plans
- Source
- Simon Willison
- Author
- Simon Willison
- Date
- eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
- human-in-the-loop — Requiring a person's approval at specific points in an automated process, chosen so the irreversible steps are the ones a human sees.
- guardrails — The checks around a model that block bad inputs and outputs — filters, validators, and permission rules the model itself can't override.
The permission default for one of the most widely used coding agents is changing on a specific date, so anyone running Claude Code needs to know what auto mode does before it becomes their default. The numbers also give a rare quantitative basis for reasoning about approval fatigue versus automated .
“We’re going to publish some evals in the coming weeks, but we’ve pretty much mitigated every attack.”
Cat Wu
“Only 13.6% of the humans refused that harmful action. Auto mode would have blocked 89% of those actions.”
“In this evaluation, none of the 720 attack attempts succeeded against Claude Fable 5, Opus 5, or Sonnet 5 running auto mode.”
“Confirmation fatigue is real, and asking humans to click "OK" every few steps is clearly not going to result in safe behavior.”
“But... I'd like to see more independent confirmation of this.”
Checking sign-in…
Loading comments…