• The benchmark isolates modality by substitution: take an existing text test like MMLU and swap individual textual concepts for semantically matched images. The question is identical, only the token type changes.
• Across 42 multimodal models the drop averages 19.6% but spans 4.2% to 47.4%, so a model's text-only score tells you almost nothing about how it will do on the same question shown as pictures.
• Neither prompting tricks nor simply scaling training compute reliably shrank the gap, which is what pushes the diagnosis away from wording or model size and toward what the model was trained on.
• The fix that did work was data: mixing image-interleaved examples into training narrowed the gap while leaving text-only and ordinary vision-language performance intact.

• multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
• LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
• benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
• token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.

If you're building or evaluating multimodal LLM applications, this reveals that models can answer the same question far less accurately when it's presented via image instead of text — a hidden failure mode worth testing for before shipping vision-language features.

• articleMultimodality and Large Multimodal Models (LMMs)Chip Huyen
• articleCapability-Driven Multimodal Scaling LawZiran Li, Qiang Wang, Zhengyu Chen, Shanglin Lei, Borun Chen, Jingang Wang, Xunliang Cai
• articleCompliance, Capability, and Conflict: Benchmarking Multimodal LLMs under System MessagesJuan Yeo, Geewook Kim

Checking sign-in…

Loading comments…