Diffusion Models for Video Generation

Generating a video requires neighboring frames to agree about the scene as it changes. Building on image diffusion, Lilian Weng's video-generation survey describes Video LDM, which adds trainable layers that exchange information across frames while retaining a pretrained image generator.

The design also addresses the stage that converts internal image representations into visible frames. A decoder trained only on individual images can introduce flicker, so Video LDM adds temporal processing there too. This explains why video adaptation involves more than producing many plausible still images: consistency must survive both the generation process and the conversion into the final sequence.

“The task itself is a superset of the image case, since an image is a video of 1 frame”