Vibeleaderboard
← All Intel
Intel / post

One reason we trust it more than manual approval: in a study with 1,053 paid tes

Source
x.com
Author
ClaudeDevs
Date

Context

Repeated permission prompts can lose their value if users stop inspecting them. In Anthropic's controlled coding study, 1,053 paid testers encountered a permission prompt whose text had been replaced with a dangerous command. Approval never executed that command. Testers rejected it 13.6% of the time, compared with auto mode blocking 89% of the same commands, according to the company's study account.

Human detection fell from about 17% early in a session to about 5% after at least 50 previous prompts. Auto mode instead evaluates proposed actions with a classifier. Anthropic presents this as a reason to trust auto mode over manual approval, though auto mode still allowed about 11% of the dangerous commands in this controlled test.

More from ClaudeDevs
Recommended reads
Comments

Checking sign-in…

Loading comments…