Modalities convert into one another: audio becomes an image as a mel spectrogram, a table becomes a chart, text becomes a photo of text. That is why image is the most versatile input format — it can carry most other modes.
Text is the most versatile output. A model that only emits images does image generation; a model that emits text does summarization, translation, reasoning, and question answering with the same weights.
Video models today mostly treat a video as a sequence of images and throw the audio away. That is a real limitation: 88% of TikTok users say sound is essential to the experience.
Text-based image search has two designs: caption every image and search the captions, or train one embeddingA list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.Full definition → space holding both text and images and match by nearest neighbor. The joint-embedding route is more flexible, with no captioning step.
multimodalA model that works with more than text — reading images, audio, or video, and sometimes generating them too.Full definition → does not mean LMM. Midjourney and Stable Diffusion take text and emit images but contain no LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition →; an LMM is specifically an LLM with extra modalities added.
Terms in this piece · Glossary
multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
grounding — Tying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
Why it matters
A clear, foundational-to-frontier walkthrough of how multimodal models actually work — from CLIP and Flamingo to modern adapter-based LMMs — giving engineers the conceptual groundingTying a model's answers to checkable sources — retrieved documents, live data, tool results — instead of letting it answer from memory alone.Full definition → to reason about and build with vision-language systems.
Key quotes
“CLIP authors found that the contrastive objective provided a 12x improvement in efficiency compared to the language model objective baseline while producing higher-quality image embeddings.”
“BLIP-2, for example, outperformed Flamingo-80B by 8.7% on zero-shot VQA-v2 with 54x fewer trainable parameters.”
“As LMMs extend upon LLMs, the performance of an LMM relies on the performance of its base LLM.”
“A model that can effectively learn from bitstrings or bytestrings will be very powerful, and it can learn from any data mode.”
articleGeneralized Visual Language ModelsLilian Weng
articleTokenSwap: Benchmarking and Reducing the Modality Gap in Multimodal LLMsAndong Hua, Colton Bishop, Igor Mordatch, Arian Hosseini, Jindong Gu, Aleksandra Faust, Rebecca Roelofs, Yao Qin
articleCapability-Driven Multimodal Scaling LawZiran Li, Qiang Wang, Zhengyu Chen, Shanglin Lei, Borun Chen, Jingang Wang, Xunliang Cai