MLLMs Fail to Refuse when Using Tools Agentically
- Source
- arxiv.org
- Author
- Rikiya Takehi, Ryo Hachiuma, Shaona Ghosh, Dan Zhao, Yu-Chiang Frank Wang, Yusuke Hirota
- Date

Context
language models, which take images as well as text, increasingly call tools such as zooming into or tagging parts of an image to reason about it. A paper accepted at NeurIPS 2026 and posted to arXiv on October 2, 2026 reports that these tool-using setups refuse harmful requests less reliably.
Testing top open and closed models on three safety benchmarks, the authors found significantly lower safety with tools than without, with the refusal failure rate rising by up to 68.7% in relative terms. Relative means scaled against the starting rate: as an illustration, a 10% failure rate would become about 16.9%. The abstract gives the worst-case increase, not a typical one.
The analysis covers more than 100,000 responses, and the authors propose two possible reasons for the drop, which the abstract does not specify. It establishes that the effect appeared across the models tested, not why.
- multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
articleCallability Is Not Operability: Controlled Interface Interventions for LLM AgentsZihao Wang
articleFrom Detection to Refusal: Safer LLMs via Circuit-Guided Weight ScalingKuan-Lin Chu, Chung-En Sun, Tsui-Wei Weng
articleForcing a Letter Suppresses Refusal: Mechanistic Evidence from WMDPStepan Epifantsev, Anna Antipova, Oleg Chernikov, Leon Strelkov, Alexey Dontsov
Checking sign-in…
Loading comments…