Diffusion models
A diffusion model generates data by learning to remove noise. It is trained to predict the noise added to real data. Then it generates by starting from pure noise and denoising step by step.
Why it matters
Generating high-quality images, audio and video is hard: GANs are unstable to train and VAEs are blurry. Diffusion models train with a simple, stable loss and produce state-of-the-art images (Stable Diffusion, DALL·E, Imagen) and video. Their main cost is slow generation, which much current research tries to cut.
The idea
A diffusion model learns to remove noise. In training, you take a real image, add a random amount of noise, and teach a network to predict that noise. To generate, you start from pure noise and remove it a little at a time, over many steps.
A sculptor who starts with a rough block and removes a little stone at each step until a statue appears.
A sculptor sees the whole plan. The model only ever learns one small clean-up step, and the full picture comes from repeating it many times. That is also why generation is slow.
How it works
- Forward process: take a real sample x₀ and add Gaussian noise. In one jump to step t: x_t = √ᾱ_t · x₀ + √(1 − ᾱ_t) · ε, where ᾱ_t falls from 1 (clean) to almost 0 (pure noise).
- Training: pick a random t, add noise ε, and train a network to predict ε from x_t and t. The loss is plain squared error.
- The network is usually a U-Net or a transformer, told the noise level t through an embedding.
- Generation: start from pure noise x_T and repeatedly use the predicted noise to take a small step towards cleaner data.
- Classic DDPM uses hundreds to a thousand steps; samplers like DDIM, and distillation, cut that to a handful.
- Conditioning: feed in a text embedding (for example from CLIP or a language model) so the denoiser follows a prompt; classifier-free guidance strengthens that effect.
- Latent diffusion runs the whole process in a VAE's compressed latent space, which is much cheaper.
Key points
- Training = predict the added noise; loss = mean squared error.
- Forward noising has a closed form for any step t.
- Generation = many small denoising steps from pure noise.
- Slow sampling is the main drawback; better samplers reduce the steps.
- Text-to-image = diffusion conditioned on a text embedding, usually in latent space.
Common mistakes
- Thinking the model removes all the noise in one step.
- Using far fewer sampling steps with a sampler not designed for it, and getting noisy images.
- Forgetting to tell the network the noise level t.
Not true: Denoising network: its input is the noisy image alone; the step t is not given.
In fact: The right correction depends on the noise level, so the step t is given to the network as an input. Without it, the network cannot tell a lightly noised image from pure noise.
Not true: Fewer sampling steps: the same sampler gives equally clean images with a tenth of the steps.
In fact: A sampler built for many small steps gives noisy images when it is cut to a few. Faster samplers are designed to work with fewer steps.
Part of Unit 7: Generative models and reinforcement learning in the free ML Foundations course.