Vision Transformer, Diffusion Transformer, and Multimodal Diffusion Transformer Explained
Source
Julia Turc
Author
Julia Turc
Date
Why it matters
Traces how the Transformer displaced CNNs in vision and became the backbone of modern image generation, explaining the conditioning mechanisms that most practitioners use without understanding.
An educational video tracing the architectural evolution from the Vision Transformer (ViT) to the Diffusion Transformer (DiT) to the Multimodal Diffusion Transformer (MMDiT), explaining how Transformers displaced CNNs as the standard architecture for vision tasks.
It covers key mechanisms like adaLN, adaLN-Zero, cross-attention, and FiLM-style conditioning, referencing the original ViT, DiT, MMDiT, and FiLM papers.
Transcript
This video covers the Vision Transformer (ViT), Diffusion Transformer (DiT) and Multimodal Diffusion Transformer (MMDiT). This is the architectural evolution that enabled the original Transformer model (initially designed for machine translation and language modeling) to replace the de-facto model for vision, the Convolutional Neural Networks (CNN). ▶️ Companion videos: - Transformers in language: https://youtu.be/SFi9KsnidNc?si=XmQpBqd0_KH7Vmcl - Diffusion fundamentals: https://youtu.be/R0uMcXsfo2o?si=LvBqX2-A1wm66iLJ - How the Transformer replaced CNNs: https://youtu.be/KnCRTP11p5U?si=2RrAya_2LU5I1Ms- 📚 Papers ViT: https://arxiv.org/abs/2010.11929 DiT: https://arxiv.org/abs/2212.09748 MMDiT: https://arxiv.org/abs/2403.03206 FiLM: https://arxiv.org/abs/1709.07871 My full reading list: https://www.patreon.com/c/JuliaTurc 00:00 Intro 01:13 Transformer recap 02:24 Image classification 03:35 Vision Transformer (ViT) 05:37 Image generation 07:54 Diffusion Transformer (DiT) 10:07 DiT in-context learning 10:38 DiT cross-attention 11:15 DiT adaLN (and FiLM inspiration) 14:26 DiT adaLN-Zero 16:03 Pixart-alpha 16:43 Multimodal Diffusion Transformer (MMDiT)
