Compliance, Capability, and Conflict: Benchmarking Multimodal LLMs under System Messages
- Source
- arxiv.org
- Author
- Juan Yeo, Geewook Kim
- Date

Context
A model that answers an image question correctly may still fail the application's instructions. VSysBench evaluates both requirements together: its paper abstract describes pairing visual tasks with system-level constraints and conflicting user requests, then scoring answer correctness alongside compliance.
Across 16 tested models, the authors report that adding system instructions reduced task accuracy and that constraints in image content were the hardest category. Under conflicting user requests, compliance deteriorated for the tested models while remaining stable for the leading proprietary models. These findings make instruction-following and visual accuracy separate requirements to test together, with the comparison limited to the evaluated models and conditions.
- grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
- open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
articleDo large language models scrutinise what they review? A multimodal audit of scoring calibration, error detection, and author-identity effectsEmad Alharbi
articleWhen Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language ModelsYinfeng Wang, Zhiyuan Yao, Zheren Fu, Lei Zhang, Zhendong Mao
articleMLLMs Fail to Refuse when Using Tools AgenticallyRikiya Takehi, Ryo Hachiuma, Shaona Ghosh, Dan Zhao, Yu-Chiang Frank Wang, Yusuke Hirota
Checking sign-in…
Loading comments…