High Resolution Image Synthesis With Latent Diffusion Models

Latent diffusion reduces the work of generating images by learning on compressed image representations instead of every pixel. The research presented by Runway uses a separate compression stage to discard visually unimportant detail, then trains the image-generation process in that smaller space. Its research summary reports competitive image-generation and editing results with lower computational requirements than pixel-based diffusion.

The amount of compression matters. Keeping every pixel makes training expensive, but compressing too aggressively loses information needed for good images. The contribution is a way to balance those costs while retaining support for instructions such as text prompts.