Weight initialisation; vanishing and exploding gradients

Weight initialisation chooses the random starting weights so that signals and gradients keep roughly the same size from layer to layer, instead of vanishing towards zero or exploding towards infinity.

Why it matters

In a deep network the signal is multiplied by a weight matrix at every layer, and the gradient again on the way back. If each layer shrinks it a little, after 50 layers nothing is left; if each grows it a little, it overflows.

Bad initialisation can make a network untrainable before it has seen a single example. Good initialisation, plus normalisation and residual connections, is what makes depth work.

The idea

If weights start too small, signals shrink layer by layer and gradients vanish. If they start too big, signals grow and explode. Good initialisation keeps the signal size steady: He initialisation for ReLU uses std = √(2 / fan_in).

Passing a message down a line of 50 people. Too quiet, and it fades to nothing. Too loud, and it turns into shouting and noise.

People can ask 'sorry, again?'. A network cannot. That is why we also add normalisation layers and residual connections.

How it works

  1. Picture one layer: each output is a sum of fan_in terms, each an input times a random weight.
  2. Adding fan_in independent terms multiplies the variance: Var(out) ≈ fan_in × Var(w) × Var(in).
  3. To keep the size the same, choose Var(w) = 1 / fan_in. This is Xavier (Glorot) initialisation, for tanh-like activations.
  4. ReLU zeroes about half its inputs, halving the variance, so double it: Var(w) = 2 / fan_in, std = √(2 / fan_in). This is He initialisation, for ReLU.
  5. Too small (std 0.01 with 48 inputs): each layer shrinks the signal, so activations and gradients vanish.
  6. Too big (std 1): each layer multiplies the signal by about √(fan_in/2), so it explodes to inf or NaN.
  7. Check it: print the standard deviation of the activations at every layer of a freshly initialised network. It should stay roughly level.

Key points

Common mistakes

Not true: A standard deviation of 1 is a safe choice for the starting weights of a wide layer.
In fact: With many inputs the sum grows at every layer and the signal explodes. He initialisation uses √(2 / fan_in), which gets smaller as the layer gets wider.

Not true: He initialisation: for tanh. Xavier initialisation: for ReLU.
In fact: It is the other way round. He, with std = √(2 / fan_in), is for ReLU, which zeroes about half its inputs. Xavier is for tanh.

Part of Unit 3: Deep learning foundations in the free ML Foundations course.