Regularisation: dropout, weight decay, augmentation, early stopping

Regularisation in deep learning is any trick that stops a network memorising its training data: dropout, weight decay, data augmentation and early stopping are the main ones.

Why it matters

Neural networks have so many parameters that they can memorise their training set exactly, even with random labels, and then fail on new data. Regularisation makes memorising harder or less attractive, so the network learns patterns that generalise. On small datasets it is often the difference between a useful model and a useless one.

The idea

Tricks to stop a network memorising. Dropout turns off random neurons during training. Weight decay shrinks weights. Data augmentation makes 'new' data, such as flipped or cropped images. Early stopping stops when the validation loss starts to rise.

Dropout is a football team that trains with random players missing, so nobody relies on one star.

Missing players happens only in training. At test time all neurons are used, so the output size must match. 'Inverted dropout' fixes this by scaling during training.

How it works

  1. Dropout: during training, zero each hidden activation with probability p (often 0.1 to 0.5) and scale the survivors by 1/(1 − p). Every step trains a different thinned network, so no neuron can rely on a particular partner.
  2. At evaluation, dropout is switched off (model.eval()), and thanks to the scaling no change is needed.
  3. Weight decay: shrink every weight a little each step (AdamW's weight_decay), pulling towards simpler functions.
  4. Data augmentation: create new, label-preserving versions of each example — flips, crops, small rotations, colour changes for images; noise or masking for audio and text.
  5. Early stopping: track validation loss during training and keep the checkpoint where it was lowest.
  6. Other levers: a smaller model, more real data, label smoothing.

Key points

Common mistakes

Not true: Validation set: augment it too, for a fairer estimate of the model.
In fact: Validation and test data should look like the real inputs. Augment the training set only, or the score no longer measures real performance.

Not true: Dropout at evaluation time: leave it switched on.
In fact: Dropout is for training only. At evaluation every neuron is kept: call model.eval(), or the predictions are noisy and worse.

Part of Unit 3: Deep learning foundations in the free ML Foundations course.