← All IntelClip / EducationLatent diffusion: compress with a VAE, diffuse in latent space
From Vision Transformer, Diffusion Transformer, and Multimodal Diffusion Transformer Explained · ≈6:30
Concrete numbers on why modern diffusion never touches raw pixels — spatial dims down to 32, channels up to 4, both forward and reverse paths entirely in latent space.
What’s in it
- Concrete numbers on why modern diffusion never touches raw pixels — spatial dims down to 32, channels up to 4, both forward and reverse paths entirely in latent space.
Clip transcript
data. The model learns to walk this chain in reverse from pure noise to a meaningful image. In practice, running diffusion in raw pixel space is expensive. That's why modern diffusion models operate in a more compact latent space. With the help of an off-the-shelf external model called the VAE or variational autoenccoder, the raw image is compressed into a smaller three-dimensional tensor. Here, the spatial dimensions were reduced to 32 to save space while the number of channels was increased from 3 to four to make room for richer features beyond just collars. So now we can encode the image with just 4,000 values. In this setup, diffusion happens entirely in latent space, both the forward and reverse paths. After reverse diffusion is complete, the final D noiseis latent is mapped back to pixel space using the decoder component from the same external off-the-shelf VAE model. During reverse diffusion, the train model is invoked repeatedly once per step to partially clean unintermediate. One meaningful consequence is that the model doesn't have a holistic view of the entire chain and operates locally. That's why the time step itself needs to be an explicit input. Otherwise, the model wouldn't know how far in the dnoising process it is. And of course, the final missing piece is a description of the desired content to generate. The original diffusion transformer was class conditional. It was trained on a fixed set of things like dogs and cats. So the user was only able to prompt it with one of these classes. We'll talk about fullyfledged text prompting a bit later. So compared to the VIT, the diffusion
Comments
Checking sign-in…
Loading comments…