Regularisation: L1 and L2
Regularisation adds a penalty for large weights to the loss. L2 shrinks every weight, while L1 pushes the weak ones to exactly zero.
Why it matters
A flexible model with many features can fit the training data too well, using huge weights that balance each other out on noise. It then does badly on new data. Regularisation tells the model "simple explanations are better": it trades a little training error for much smaller weights, which usually generalise better. L1 also picks features for you.
The idea
Add a penalty for large weights to the loss. L2 adds λΣw² and shrinks all weights a little. L1 adds λΣ|w| and pushes some weights to exactly zero.
A spending budget. L2 is a tax that grows fast for big spending, so you spend a little on everything. L1 is a flat fee per pound, so it is often best to spend nothing at all on weak items.
The 'cost' here is not money but model complexity. The right λ must be found with validation data, not guessed.
How it works
- Start from the normal loss, for example mean squared error.
- Add a penalty: L2 adds λΣw², L1 adds λΣ|w|. λ (lambda) sets how strongly simplicity is preferred.
- L2's gradient adds 2λw to each weight's gradient, so every step also shrinks each weight a little, in proportion to its size. This is "weight decay".
- With uncorrelated, equally scaled features, L2 simply divides every weight by (1 + λ): smaller, but never exactly zero.
- L1 pulls every weight towards zero by the same fixed amount, λ/2 in that simple case. Weights smaller than that land on exactly 0.
- Choose λ with the validation set: try values like 0.001, 0.01, 0.1, 1 and 10, and keep the one with the best validation score.
- Never penalise the bias, and scale the features first, so the penalty treats them fairly.
Key points
- Loss = data error + λ × penalty. λ = 0 means no regularisation.
- L2 (ridge, weight decay): small weights everywhere; smooth; has a closed form w = (XᵀX + λI)⁻¹Xᵀy.
- L1 (lasso): many weights exactly 0, so it selects features.
- Too big a λ underfits; too small overfits. Tune it on validation data.
- Regularisation reduces variance at the cost of a little bias.
- In deep learning, weight decay, dropout, data augmentation and early stopping all play this role (Unit 3).
Common mistakes
- Regularising without scaling features: features measured in small units need big weights, so they get punished unfairly.
- Penalising the bias term as well. Regularise only the weights: shrinking the bias just shifts every prediction.
- Picking λ by looking at the test set.
- Expecting L2 to remove features. It only shrinks them.
Not true: A penalty on the weights treats every feature fairly, whatever units the features are measured in.
In fact: A feature measured in small units needs a big weight, so the penalty punishes it more. Scale the features before regularising.
Not true: L2 regularisation removes unneeded features by setting their weights exactly to 0.
In fact: L2 only shrinks weights towards 0; they rarely reach it. L1 is the penalty that sets weights exactly to 0.
Go deeper: Why L2 regularisation is a Gaussian prior on the weights
- Maximum likelihood picks the weights that make the data most probable: maximise P(data | w). MAP also uses a belief about the weights before seeing data, the prior P(w): maximise P(data | w)·P(w).
- Take logs and flip the sign, so that we minimise: −log P(data | w) − log P(w). With Gaussian noise of variance σ², the first term is the squared error: Σ (y − ŷ)² / (2σ²).
- Choose the prior "weights are probably small": each weight is Gaussian with mean 0 and variance τ². Then −log P(w) = Σ w² / (2τ²), plus a constant.
- Add the two and multiply through by 2σ²: the objective is Σ (y − ŷ)² + (σ²/τ²)·Σ w².
- That is squared error plus an L2 penalty, with λ = σ²/τ². A stronger belief that weights are small (small τ), or noisier data (large σ), means a larger λ.
- Use a Laplace prior instead, whose −log is |w|/b, and the penalty becomes λΣ|w|: L1.
L2 regularisation is MAP estimation with a Gaussian prior on the weights, with λ = σ²/τ². L1 is MAP with a Laplace prior.
Part of Unit 2: Classical ML, built by you in the free ML Foundations course.