Autoencoders and VAEs (the ELBO)

An autoencoder squeezes data into a small code and rebuilds it. A variational autoencoder (VAE) makes the code a probability distribution, pulled towards a simple bell curve, so that you can sample new codes and decode them into new data.

Why it matters

A plain autoencoder learns good compression, but its codes are scattered with gaps, so a random code decodes into nonsense and it can't generate anything new. VAEs fix that by shaping the code space, giving a principled generative model with a smooth latent space. The same encoder–decoder idea is inside modern image generators, which run diffusion in a VAE's compact latent space.

The idea

An autoencoder squeezes data into a small code and rebuilds it. A variational autoencoder (VAE) makes that code a probability distribution, so you can sample new codes and decode them into new data. It trains by maximising the ELBO: rebuild the input well, but keep the codes close to a simple bell curve.

Describing a face to a sketch artist in 10 numbers. The encoder writes the description and the decoder draws. A VAE also forces the descriptions to be tidy, so a random description still gives a sensible face.

A person's description is exact. A VAE's code is fuzzy on purpose (it adds noise), which is one reason VAE images are often blurry.

How it works

  1. Encoder: a network maps the input x to the mean μ and spread σ of a distribution over codes z.
  2. Sample a code with the reparameterisation trick: z = μ + σ·ε with ε ~ N(0, I). The randomness is in ε, so gradients can flow through μ and σ.
  3. Decoder: a network maps z back to a reconstruction of x.
  4. The loss is the negative ELBO: a reconstruction term (rebuild x well) plus a KL term that pulls each q(z | x) towards the prior N(0, I).
  5. The KL term packs codes into one smooth, gap-free cloud; the reconstruction term keeps them informative.
  6. To generate: sample z from N(0, I) and decode it.
  7. β-VAE weights the KL term by β: bigger β gives a tidier, more disentangled space but blurrier reconstructions.

Key points

Common mistakes

Not true: Basic VAE samples: crisp and photo-quality.
In fact: They tend to be blurry. The code is fuzzy on purpose (noise is added to it), so the decoder learns to draw an average of the images that fit. Diffusion models give sharper images.

Not true: Sampling z at random: gradients flow straight through that step.
In fact: Sampling is not differentiable. The reparameterisation trick writes z = μ + σ·ε, with the randomness in ε, so gradients can reach μ and σ.

Part of Unit 7: Generative models and reinforcement learning in the free ML Foundations course.