Optimisers: SGD, momentum, Adam; learning-rate schedules

An optimiser turns gradients into weight updates. SGD steps straight down the gradient, momentum adds velocity, and Adam also adapts the step size for every weight. A learning-rate schedule changes the step size during training.

Why it matters

Real loss surfaces have long narrow valleys, flat plateaus and noisy gradients from small batches. Plain SGD zig-zags across steep valleys and crawls along flat ones. Better optimisers converge in far fewer steps and are much less fussy about the learning rate, which saves days of compute on large models.

The idea

SGD steps against the gradient. Momentum keeps a running average of gradients, so you keep moving in a consistent direction. Adam adds a separate step size for each weight, based on how large its gradients usually are. A schedule changes the learning rate over time, for example warm-up then decay.

SGD is a hiker who only looks at the ground under their feet. Momentum is a ball rolling downhill: it builds speed and rolls over small bumps. Adam is a ball that is careful in steep, bumpy directions and bolder in flat ones.

A ball has one mass. Adam gives every single weight its own effective step size, so it is really millions of different balls.

How it works

  1. SGD: w ← w − η·g, where g is the gradient on a mini-batch.
  2. Momentum: keep a velocity v ← β·v + g (β ≈ 0.9) and step w ← w − η·v. Side-to-side gradients cancel out while consistent ones build up speed.
  3. RMSProp: keep a running average of g² and divide each step by its square root, so directions with big gradients take smaller steps.
  4. Adam combines both: m ← β₁m + (1 − β₁)g and v ← β₂v + (1 − β₂)g², then w ← w − η·m̂ / (√v̂ + ε), with β₁ = 0.9 and β₂ = 0.999.
  5. Bias correction: m and v start at 0, so early on they are too small. Dividing by (1 − βᵗ) fixes this in the first steps.
  6. AdamW applies weight decay directly to the weights instead of through the gradient. It is the standard for transformers.
  7. Schedules: warm up from a tiny learning rate over the first steps, then decay it (cosine or linear) so training settles into a good minimum.

Key points

Common mistakes

Not true: A learning rate that works for Adam works just as well for plain SGD.
In fact: Adam rescales every step, so its usual rates (around 0.001) are far smaller than SGD's (around 0.1). Tune the learning rate separately for each optimiser.

Not true: With Adam, adding an L2 term to the loss is exactly the same as weight decay.
In fact: Adam rescales every gradient, the L2 part included, so the weights are not shrunk evenly. AdamW shrinks the weights in a separate step, which is the correct weight decay.

Part of Unit 3: Deep learning foundations in the free ML Foundations course.