Vibeleaderboard
← All Intel
Intel / article

Compliance, Capability, and Conflict: Benchmarking Multimodal LLMs under System Messages

Source
arxiv.org
Author
Juan Yeo, Geewook Kim
Date

Context

A model that answers an image question correctly may still fail the application's instructions. VSysBench evaluates both requirements together: its paper abstract describes pairing visual tasks with system-level constraints and conflicting user requests, then scoring answer correctness alongside compliance.

Across 16 tested models, the authors report that adding system instructions reduced task accuracy and that constraints in image content were the hardest category. Under conflicting user requests, compliance deteriorated for the tested models while remaining stable for the leading proprietary models. These findings make instruction-following and visual accuracy separate requirements to test together, with the comparison limited to the evaluated models and conditions.

Terms in this piece · Glossary
  • grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
  • open weights — A model whose trained parameters are published for anyone to download and run — unlike API-only models you can access but never possess.
  • benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
  • multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
Recommended reads
Comments

Checking sign-in…

Loading comments…