Unit 7: Generative models and reinforcement learning
Models that create new data, and models that learn by trial and error. These ideas power image generators, and the reinforcement learning behind RLHF.
The big picture
Every model so far learned to predict a label. Generative models learn what the data itself looks like, so they can make new examples. Reinforcement learning has no labels at all, only rewards, so the model must discover good actions by trying.
Lessons in this unit
- Autoencoders and VAEs (the ELBO): An autoencoder squeezes data into a small code and rebuilds it. A variational autoencoder (VAE) makes the code a probability distribution, pulled towards a simple bell curve, so that you can sample new codes and decode them into new data.
- GANs: a generator against a discriminator: A generative adversarial network (GAN) trains two networks against each other: a generator that turns random noise into fake samples, and a discriminator that tries to tell fake from real. Each improves by beating the other.
- Diffusion models: A diffusion model generates data by learning to remove noise. It is trained to predict the noise added to real data. Then it generates by starting from pure noise and denoising step by step.
- Reinforcement learning basics: states, actions, rewards, values: In reinforcement learning, an agent takes actions in an environment, receives rewards, and learns a policy — what to do in each state — that maximises the total (discounted) reward over time.
- Deep Q-networks (DQN): Q-learning with a neural network: A deep Q-network replaces the Q-table of Q-learning with a neural network that takes a state and outputs a value for every action. It is trained on remembered experience, towards targets computed by a slowly updated copy of itself.
- Policy gradients and PPO: Policy-gradient methods train a policy network directly by making actions that led to better-than-expected returns more likely; PPO adds a clipped objective that keeps each update small and safe.