Optimisers: SGD, momentum, Adam; learning-rate schedules
An optimiser turns gradients into weight updates. SGD steps straight down the gradient, momentum adds velocity, and Adam also adapts the step size for every weight. A learning-rate schedule changes the step size during training.
Why it matters
Real loss surfaces have long narrow valleys, flat plateaus and noisy gradients from small batches. Plain SGD zig-zags across steep valleys and crawls along flat ones. Better optimisers converge in far fewer steps and are much less fussy about the learning rate, which saves days of compute on large models.
The idea
SGD steps against the gradient. Momentum keeps a running average of gradients, so you keep moving in a consistent direction. Adam adds a separate step size for each weight, based on how large its gradients usually are. A schedule changes the learning rate over time, for example warm-up then decay.
SGD is a hiker who only looks at the ground under their feet. Momentum is a ball rolling downhill: it builds speed and rolls over small bumps. Adam is a ball that is careful in steep, bumpy directions and bolder in flat ones.
A ball has one mass. Adam gives every single weight its own effective step size, so it is really millions of different balls.
How it works
- SGD: w ← w − η·g, where g is the gradient on a mini-batch.
- Momentum: keep a velocity v ← β·v + g (β ≈ 0.9) and step w ← w − η·v. Side-to-side gradients cancel out while consistent ones build up speed.
- RMSProp: keep a running average of g² and divide each step by its square root, so directions with big gradients take smaller steps.
- Adam combines both: m ← β₁m + (1 − β₁)g and v ← β₂v + (1 − β₂)g², then w ← w − η·m̂ / (√v̂ + ε), with β₁ = 0.9 and β₂ = 0.999.
- Bias correction: m and v start at 0, so early on they are too small. Dividing by (1 − βᵗ) fixes this in the first steps.
- AdamW applies weight decay directly to the weights instead of through the gradient. It is the standard for transformers.
- Schedules: warm up from a tiny learning rate over the first steps, then decay it (cosine or linear) so training settles into a good minimum.
Key points
- SGD: simple, and with momentum and a good schedule still excellent for vision models.
- Momentum smooths noisy gradients and speeds up travel along valleys.
- Adam gives each parameter its own effective step size; it is robust to the learning rate.
- Typical starting learning rates: SGD ≈ 0.1, Adam ≈ 0.001 (3e-4 is a famous default).
- Warm-up prevents huge early steps while the weights and Adam's averages are still unreliable.
- Adam stores two extra numbers per weight, which matters for memory in large models (Unit 6).
Common mistakes
- Using Adam's learning rate with SGD, or the other way round.
- Leaving out warm-up for transformers, which often makes early training diverge.
- Adding L2 to the loss with Adam and calling it weight decay. Use AdamW.
- Comparing optimisers without tuning each one's learning rate.
Not true: A learning rate that works for Adam works just as well for plain SGD.
In fact: Adam rescales every step, so its usual rates (around 0.001) are far smaller than SGD's (around 0.1). Tune the learning rate separately for each optimiser.
Not true: With Adam, adding an L2 term to the loss is exactly the same as weight decay.
In fact: Adam rescales every gradient, the L2 part included, so the weights are not shrunk evenly. AdamW shrinks the weights in a separate step, which is the correct weight decay.
Part of Unit 3: Deep learning foundations in the free ML Foundations course.