Batch norm and layer norm

A normalisation layer rescales activations to mean 0 and variance 1, then lets the network scale and shift them again. Batch norm does this for each feature across the batch; layer norm does it for each example across its features.

Why it matters

As earlier layers learn, the values reaching later layers keep changing scale, so each layer chases a moving target and training needs tiny learning rates. Normalisation keeps activations in a steady range, which allows bigger learning rates, makes deep networks far easier to train, and reduces sensitivity to initialisation. Every modern architecture uses one of these layers.

The idea

Normalisation rescales activations to mean 0 and variance 1, then lets the network rescale them with learned γ and β. Batch norm averages over the batch, for each feature. Layer norm averages over the features, for each example.

Grading on a curve. Batch norm curves each exam question across all students. Layer norm curves each student across all of their questions.

A curve is set once. Batch norm behaves differently at test time (it uses saved running averages), which is a common source of bugs. Layer norm does not depend on the batch, which is why transformers use it.

How it works

  1. Take a batch of activations with shape (batch, features).
  2. Batch norm: for each feature (column), compute its mean and variance over the batch.
  3. Subtract the mean and divide by √(variance + ε): every column now has mean 0 and spread 1.
  4. Multiply by a learned γ and add a learned β, so the network can undo the normalisation if that helps.
  5. At test time, batch norm uses running averages of the mean and variance saved during training, so model.eval() matters.
  6. Layer norm: for each example (row), use the mean and variance over its own features. No batch statistics, so training and test behave the same, and batch size doesn't matter.
  7. Transformers use layer norm (or RMSNorm, which skips subtracting the mean). CNNs traditionally use batch norm.

Key points

Common mistakes

Not true: Layer norm needs a large batch to work well.
In fact: Layer norm uses only the features of one example, so the batch size makes no difference. It is batch norm that needs a reasonable batch.

Not true: Batch norm behaves the same in training and at test time.
In fact: In training it uses the statistics of the current batch. At test time it uses running averages saved during training: call model.eval() to switch.

Part of Unit 3: Deep learning foundations in the free ML Foundations course.