Compliance, Capability, and Conflict: Benchmarking Multimodal LLMs under System Messages
Source
arxiv.org
Author
Juan Yeo, Geewook Kim
Date
Why it matters
Anyone putting constraints in a system promptThe standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.Full definition → for a vision model is trading away task accuracy without knowing the rate.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
system prompt — The standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.
articleDo large language models scrutinise what they review? A multimodal audit of scoring calibration, error detection, and author-identity effectsEmad Alharbi
articleWhen Irrelevant Text Matters: Affine Margin Shifts in Multimodal Large Language ModelsYinfeng Wang, Zhiyuan Yao, Zheren Fu, Lei Zhang, Zhendong Mao
articleInter-dimension Dependence for Multi-Dimensional Evaluation of Open-Ended TextHaoyuan Li, Snigdha Chaturvedi