← All IntelClip / EducationWhy vision transformers keep only the encoder
From Vision Transformer, Diffusion Transformer, and Multimodal Diffusion Transformer Explained · ≈1:53
Explains a design choice that is otherwise arbitrary: causal left-to-right attention mirrors reading, but images are processed holistically, so bidirectional encoding wins.
What’s in it
- Explains a design choice that is otherwise arbitrary: causal left-to-right attention mirrors reading, but images are processed holistically, so bidirectional encoding wins.
Clip transcript
reggressively. Its attention is causal. Each output token can only depend on the ones that came before and cannot peak into the future. While left to right processing is intuitive for language since it mirrors how we read and process text, the same framing doesn't translate well to vision. As humans, we process images holistically where every part can inform every other part. So birectional attention is a more natural choice. That's why most modern vision transformers only borrow the encoder component. Let's start with the simplest vision
Comments
Checking sign-in…
Loading comments…