Compliance, Capability, and Conflict: Benchmarking Multimodal LLMs under System Messages
Source
Juan Yeo, Geewook Kim
Author
Juan Yeo, Geewook Kim
Date
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
system prompt — The standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.
Why it matters
Anyone putting constraints in a system promptThe standing instructions a model receives before any user input — defining its role, rules, tools, and tone for the whole conversation.Full definition → for a vision model is trading away task accuracy without knowing the rate.
articleDiagnostic Foundation for Evaluating LLMs' Research Integrity as Co-ScientistsYash Tripathi, Silu Sharma, Sai Sidhanth Manoharan Jayanthi, Shivank Garg, Lin Li
articleThe Unwritten Benchmark: A New Challenge for Multimodal Machine Learning in Abstract Perceptual ReasoningGarima Arya Yadav, Nilay Yilmaz, Yezhou Yang
articleSelf- and Other-Labels Induce Bidirectional Bias in LLM JudgesSongeun Chae, Min Kim, Donghoon Jung, Seojin Choi, Seohyon Jung