Gradient descent and the learning rate

Gradient descent makes a function smaller by taking many small steps downhill, against its gradient. The learning rate sets the size of each step.

Why it matters

For most models there is no formula that gives the best weights directly, and for neural networks there certainly isn't. But you can always compute the gradient. Gradient descent turns "which way is downhill?" into an algorithm that walks to a minimum one small step at a time. Choosing the step size well is the difference between fast training, slow training and no training at all.

The idea

To make a loss smaller, take a small step against the gradient (downhill), and repeat. The learning rate is the size of each step.

Walking down a mountain in thick fog. You feel the slope under your feet and step downhill.

In fog you might stop in a small valley that is not the lowest one. In very high dimensions, flat 'saddle' areas are usually the bigger problem.

How it works

  1. Pick a starting point: w = 4, where f(w) = w² = 16.
  2. Compute the gradient there: f′(w) = 2w = 8. It is positive, so downhill is to the left.
  3. Step against it, scaled by the learning rate: w ← 4 − 0.1·8 = 3.2.
  4. Repeat. Each step here multiplies w by (1 − 2η) = 0.8: 4 → 3.2 → 2.56 → 2.05 → … After 50 steps w ≈ 0.00006.
  5. The steps shrink by themselves near the bottom, because the gradient shrinks there.
  6. With η = 0.9 the factor is 1 − 1.8 = −0.8: w flips sign every step but still shrinks, a zig-zag that converges.
  7. With η = 1.1 the factor is −1.2: every step overshoots further than the last, and w explodes: 4, −4.8, 5.76, −6.91, …

Key points

Common mistakes

Not true: Gradient descent works equally well whether or not the features are on similar scales.
In fact: Features on very different scales turn the loss surface into a long, thin valley, where gradient descent zig-zags and crawls. Scale them first.

Not true: A bigger learning rate always reaches the minimum sooner.
In fact: Too big a step jumps over the minimum: the weight swings from side to side, or flies away. For f(w) = w², any η above 1 diverges.

Go deeper: Why the learning rate has a hard upper limit

  1. Take a bowl of any steepness: f(w) = (a/2)·w², with a > 0. Its gradient is a·w, and a measures how sharply the bowl curves (it is the second derivative).
  2. One step of gradient descent: w ← w − η·a·w = (1 − ηa)·w. Every step multiplies w by the same number, 1 − ηa.
  3. After k steps, w = (1 − ηa)ᵏ·w₀. This shrinks to 0 exactly when |1 − ηa| < 1.
  4. Solve it: −1 < 1 − ηa < 1 gives 0 < η < 2/a. The lesson's f(w) = w² has a = 2, so the limit is η < 1: that is why 0.9 converges and 1.1 explodes.
  5. The fastest choice is η = 1/a: the factor is 0 and one step lands on the minimum. Between 1/a and 2/a the factor is negative, so w overshoots and zig-zags but still converges.
  6. With many weights, each direction has its own curvature. η must stay below 2 / (the largest curvature), which makes progress slow in the flattest direction. That gap is why badly scaled problems train slowly, and why momentum and Adam help.

On a bowl with curvature a, gradient descent converges only for 0 < η < 2/a, and η = 1/a is the fastest. The sharpest direction sets the limit for all the others.

Part of Unit 1: Math foundations in the free ML Foundations course.