Autoencoders and VAEs (the ELBO)
An autoencoder squeezes data into a small code and rebuilds it. A variational autoencoder (VAE) makes the code a probability distribution, pulled towards a simple bell curve, so that you can sample new codes and decode them into new data.
Why it matters
A plain autoencoder learns good compression, but its codes are scattered with gaps, so a random code decodes into nonsense and it can't generate anything new. VAEs fix that by shaping the code space, giving a principled generative model with a smooth latent space. The same encoder–decoder idea is inside modern image generators, which run diffusion in a VAE's compact latent space.
The idea
An autoencoder squeezes data into a small code and rebuilds it. A variational autoencoder (VAE) makes that code a probability distribution, so you can sample new codes and decode them into new data. It trains by maximising the ELBO: rebuild the input well, but keep the codes close to a simple bell curve.
Describing a face to a sketch artist in 10 numbers. The encoder writes the description and the decoder draws. A VAE also forces the descriptions to be tidy, so a random description still gives a sensible face.
A person's description is exact. A VAE's code is fuzzy on purpose (it adds noise), which is one reason VAE images are often blurry.
How it works
- Encoder: a network maps the input x to the mean μ and spread σ of a distribution over codes z.
- Sample a code with the reparameterisation trick: z = μ + σ·ε with ε ~ N(0, I). The randomness is in ε, so gradients can flow through μ and σ.
- Decoder: a network maps z back to a reconstruction of x.
- The loss is the negative ELBO: a reconstruction term (rebuild x well) plus a KL term that pulls each q(z | x) towards the prior N(0, I).
- The KL term packs codes into one smooth, gap-free cloud; the reconstruction term keeps them informative.
- To generate: sample z from N(0, I) and decode it.
- β-VAE weights the KL term by β: bigger β gives a tidier, more disentangled space but blurrier reconstructions.
Key points
- Autoencoder: compress and rebuild; VAE: also a generative model.
- A VAE is trained by maximising the ELBO = reconstruction − KL(q(z | x) ‖ p(z)): rebuild the input well, and keep the codes close to the prior.
- The reparameterisation trick makes sampling differentiable.
- KL too weak → gaps; too strong → codes ignore the input ("posterior collapse").
- VAE samples tend to be blurry; diffusion and other models give sharper images.
- Latent spaces support interpolation: walk between two codes and decode.
Common mistakes
- Sampling z directly without the reparameterisation trick, so gradients can't flow.
- Letting the KL term dominate early, so the decoder learns to ignore z.
- Expecting crisp, photo-quality samples from a basic VAE.
Not true: Basic VAE samples: crisp and photo-quality.
In fact: They tend to be blurry. The code is fuzzy on purpose (noise is added to it), so the decoder learns to draw an average of the images that fit. Diffusion models give sharper images.
Not true: Sampling z at random: gradients flow straight through that step.
In fact: Sampling is not differentiable. The reparameterisation trick writes z = μ + σ·ε, with the randomness in ε, so gradients can reach μ and σ.
Part of Unit 7: Generative models and reinforcement learning in the free ML Foundations course.