Diffusion models

A diffusion model generates data by learning to remove noise. It is trained to predict the noise added to real data. Then it generates by starting from pure noise and denoising step by step.

Why it matters

Generating high-quality images, audio and video is hard: GANs are unstable to train and VAEs are blurry. Diffusion models train with a simple, stable loss and produce state-of-the-art images (Stable Diffusion, DALL·E, Imagen) and video. Their main cost is slow generation, which much current research tries to cut.

The idea

A diffusion model learns to remove noise. In training, you take a real image, add a random amount of noise, and teach a network to predict that noise. To generate, you start from pure noise and remove it a little at a time, over many steps.

A sculptor who starts with a rough block and removes a little stone at each step until a statue appears.

A sculptor sees the whole plan. The model only ever learns one small clean-up step, and the full picture comes from repeating it many times. That is also why generation is slow.

How it works

  1. Forward process: take a real sample x₀ and add Gaussian noise. In one jump to step t: x_t = √ᾱ_t · x₀ + √(1 − ᾱ_t) · ε, where ᾱ_t falls from 1 (clean) to almost 0 (pure noise).
  2. Training: pick a random t, add noise ε, and train a network to predict ε from x_t and t. The loss is plain squared error.
  3. The network is usually a U-Net or a transformer, told the noise level t through an embedding.
  4. Generation: start from pure noise x_T and repeatedly use the predicted noise to take a small step towards cleaner data.
  5. Classic DDPM uses hundreds to a thousand steps; samplers like DDIM, and distillation, cut that to a handful.
  6. Conditioning: feed in a text embedding (for example from CLIP or a language model) so the denoiser follows a prompt; classifier-free guidance strengthens that effect.
  7. Latent diffusion runs the whole process in a VAE's compressed latent space, which is much cheaper.

Key points

Common mistakes

Not true: Denoising network: its input is the noisy image alone; the step t is not given.
In fact: The right correction depends on the noise level, so the step t is given to the network as an input. Without it, the network cannot tell a lightly noised image from pure noise.

Not true: Fewer sampling steps: the same sampler gives equally clean images with a tenth of the steps.
In fact: A sampler built for many small steps gives noisy images when it is cut to a few. Faster samplers are designed to work with fewer steps.

Part of Unit 7: Generative models and reinforcement learning in the free ML Foundations course.