Intel / article

TokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMs

Source: arxiv.org
Author: Andong Hua, Colton Bishop, Igor Mordatch, Arian Hosseini, Jindong Gu, Aleksandra Faust, Rebecca Roelofs, Yao Qin
Date: Published Aug 3, 2026

Context

An AI model can answer a written question successfully yet struggle when equivalent information appears partly as pictures. TokenSwap tests that difference by replacing concepts in text benchmarks with corresponding images, leaving text and pictures interleaved. Across 42 models, the researchers report lower performance on these mixed inputs. They also report that training with TokenSwap reduced the gap while preserving text-only and vision-language performance. The finding identifies a specific weakness in combining words and images: success on the text version does not establish success when the same meaning is presented visually.

Key takeaways · AI-distilled

The isolates modality by substitution: take an existing text test like MMLU and swap individual textual concepts for semantically matched images. The question is identical, only the type changes.
Across 42 models the drop averages 19.6% but spans 4.2% to 47.4%, so a model's text-only score tells you almost nothing about how it will do on the same question shown as pictures.
Neither prompting tricks nor simply scaling training compute reliably shrank the gap, which is what pushes the diagnosis away from wording or model size and toward what the model was trained on.
The fix that did work was data: mixing image-interleaved examples into training narrowed the gap while leaving text-only and ordinary vision-language performance intact.

Terms in this piece · Glossary

multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.

Key quotes

“we observe a pervasive modality gap, with performance decreasing by 4.2% to 47.4% when moving from text-only to image-interleaved inputs, averaging 19.6% +/- 3.3% across models”

“reasoning models exhibit consistently smaller gaps, achieving an average gap of 10.1% compared to 25.5% for non-reasoning models”

“neither prompting strategies nor scaling training compute alone reliably reduces the modality gap”

Read the source arxiv.org