Policy gradients and PPO
Policy-gradient methods train a policy network directly by making actions that led to better-than-expected returns more likely; PPO adds a clipped objective that keeps each update small and safe.
Why it matters
Value-based methods like Q-learning struggle with huge or continuous action spaces — and a language model's "action" is a choice among tens of thousands of tokens. Policy gradients optimise the policy itself and handle such spaces naturally. PPO made them stable enough to use widely, including in classic RLHF for chat models.
The idea
Policy-gradient methods train the policy network directly: make actions that led to high reward more likely, and actions that led to low reward less likely. PPO adds a safety limit so each update cannot change the policy too much. PPO is the algorithm used in classic RLHF for language models.
A basketball player adjusting their shot: after a good shot, do a bit more of that; after a miss, a bit less. PPO is a coach who says 'small changes only'.
A player can feel which part of the shot went wrong. Policy gradients only see the total reward, so the signal is very noisy and needs many tries. Baselines and advantages reduce that noise.
How it works
- The policy π(a | s) is a network that outputs a probability for each action (for an LLM, each next token).
- Collect episodes by acting with the current policy and record the returns.
- Advantage A = how much better the outcome was than expected (return minus a baseline, such as a learned value estimate).
- REINFORCE update: increase log π(a | s) in proportion to A. Better-than-expected actions become more likely; worse ones less likely.
- Subtracting a baseline doesn't change the expected direction but greatly reduces noise.
- PPO measures the ratio r = π_new(a | s) / π_old(a | s) and maximises min(r·A, clip(r, 1 − ε, 1 + ε)·A), with ε around 0.2.
- Clipping removes any reward for moving the policy beyond the trust range in one update, so the same batch can be reused for several small, stable steps.
Key points
- Optimise the policy directly: ∇J ≈ E[∇log π(a | s) · A].
- Subtracting a baseline from the return, which gives the advantage, reduces the variance of the gradient without changing its average.
- PPO's clip keeps the new policy close to the old one.
- On-policy: data must come from (nearly) the current policy.
- In RLHF, the "environment" is a reward model, and a KL penalty keeps the model near its starting point.
Common mistakes
- Using no baseline, so the gradient estimate is extremely noisy and learning is slow.
- Huge steps without clipping, so the policy collapses.
- Reusing very old data with an on-policy method.
Not true: An on-policy method such as PPO can keep learning from data collected by a much older policy.
In fact: On-policy methods assume the data came from (nearly) the current policy. Very old data describes a different policy, so the updates it gives are wrong.
Not true: Baseline: subtracting it changes the direction of the policy gradient.
In fact: A baseline changes only the noise, not the average direction. The gradient points the same way with far less variance, which is why the advantage is used.
Part of Unit 7: Generative models and reinforcement learning in the free ML Foundations course.