Intel / article

Multimodality and Large Multimodal Models (LMMs)

Source: huyenchip.com
Author: Chip Huyen
Date: Published Oct 10, 2023

Context

Searching pictures with words requires a way to compare two different kinds of input. In Chip Huyen's multimodality guide, CLIP illustrates one solution: convert images and text into numerical representations that can be compared in a shared space. A written query can then retrieve images with nearby representations.

The CLIP paper explains how training creates that connection: matching image-text pairs receive higher similarity scores, while mismatched pairs receive lower ones. This teaches a model to match descriptions to pictures. Generating an answer about a picture requires an additional text-generation mechanism, so image retrieval and visual conversation are distinct capabilities.

Key takeaways · AI-distilled

Modalities convert into one another: audio becomes an image as a mel spectrogram, a table becomes a chart, text becomes a photo of text. That is why image is the most versatile input format — it can carry most other modes.
Text is the most versatile output. A model that only emits images does image generation; a model that emits text does summarization, translation, reasoning, and question answering with the same weights.
Video models today mostly treat a video as a sequence of images and throw the audio away. That is a real limitation: 88% of TikTok users say sound is essential to the experience.
Text-based image search has two designs: caption every image and search the captions, or train one space holding both text and images and match by nearest neighbor. The joint-embedding route is more flexible, with no captioning step.
does not mean LMM. Midjourney and Stable Diffusion take text and emit images but contain no ; an LMM is specifically an LLM with extra modalities added.

Terms in this piece · Glossary

multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
embedding — A list of numbers representing a piece of text's meaning, so that similar meanings end up numerically close and can be searched.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.

Key quotes

“CLIP authors found that the contrastive objective provided a 12x improvement in efficiency compared to the language model objective baseline while producing higher-quality image embeddings.”

“BLIP-2, for example, outperformed Flamingo-80B by 8.7% on zero-shot VQA-v2 with 54x fewer trainable parameters.”

“As LMMs extend upon LLMs, the performance of an LMM relies on the performance of its base LLM.”

“A model that can effectively learn from bitstrings or bytestrings will be very powerful, and it can learn from any data mode.”

Read the source huyenchip.com