Stanford CS25: Transformers United V6 I From Language Models to Native Multimodal Intelligence
Source
youtube.com
Author
Stanford Online
Date
Context
A model that understands images does not necessarily generate images. This CS25 lecture distinguishes systems that accept several kinds of input but produce text from systems trained to generate multiple kinds of output.
The shared building idea is to convert different inputs into sequences the model can process. Images can become encoded patches, and video can become sequences of encoded frames. What the model is trained to predict then determines its output capabilities. The speaker treats input support and output support as separate questions, noting that little work shows training a model to generate images improves its image understanding.
Terms in this piece · Glossary
multimodal — A model that works with more than text — reading images, audio, or video, and sometimes generating them too.
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.