Bias–variance trade-off and overfitting
Bias is error from a model too simple to capture the pattern. Variance is error from a model so flexible that it chases the noise in its training data. Good models balance the two.
Why it matters
Every model choice — degree, depth, number of features, k — moves you between underfitting and overfitting. Without this picture you can't tell whether to make a model bigger or smaller, or whether to collect more data. The bias–variance trade-off is the map for those decisions.
The idea
Bias is error from a model that is too simple. Variance is error from a model that is too sensitive to the training data. Overfitting means low training error but high test error.
Two students. One memorises every answer and fails on new questions (high variance). One only learns 'the answer is usually C' (high bias). You want the one who learns the ideas.
With very large deep learning models, more size can sometimes reduce overfitting ('double descent'). The simple trade-off picture is not the whole story.
How it works
- Take 12 noisy points from a smooth curve, and fit polynomials of growing degree.
- Degree 1 (a line) can't bend: it misses the shape on training and validation data alike. That is high bias, or underfitting.
- Degree 3 follows the real shape: training and validation errors are both low.
- Degree 9 bends through almost every training point, but swings wildly in between: tiny training error, large validation error. That is high variance, or overfitting.
- Plot both errors against degree. Training error keeps falling; validation error falls and then rises, a U shape. Choose the bottom of the U.
- More data reduces variance, because a flexible model has less room to chase noise. A simpler model or regularisation reduces it too.
- Formally: expected error = bias² + variance + irreducible noise. You can trade the first two, but never beat the noise.
Key points
- High bias: bad on training data too. Fix: a bigger model, better features, less regularisation.
- High variance: good on training data, bad on validation. Fix: more data, regularisation, a simpler model, early stopping.
- Training error alone can't tell you which problem you have. Compare it with validation error.
- Learning curves (error against training-set size) also show it: a big, stable gap means variance.
- Very large neural networks can break the simple U shape ("double descent", later in this unit).
Common mistakes
- Picking the model with the lowest training error.
- Adding complexity when the model is already overfitting.
- Collecting more data to fix bias. It won't help a model that is too simple.
Not true: Collecting more data is the reliable fix for a model that is too simple.
In fact: More data reduces variance, not bias. A model too simple for the pattern stays wrong however much data it sees: make the model or its features richer.
Not true: The model with the lowest training error is the best one to choose.
In fact: A model can reach a low training error by memorising. Choose by validation error, which shows how it does on data it has not seen.
Go deeper: Proving that expected error = bias² + variance + noise
- Set-up: the data follows y = f(x) + ε, where the noise ε has mean 0 and variance σ². A model trained on a random training set predicts f̂(x). Fix one point x. Write f for the true value, f̂ for the prediction, and m = E[f̂] for the prediction averaged over every possible training set.
- Split the error into three pieces: y − f̂ = ε + (f − m) + (m − f̂). They are the noise, how far the average model is from the truth, and how far this model is from the average model.
- Square it and take the expectation. The three squares give E[ε²] = σ², (f − m)² and E[(m − f̂)²].
- The cross terms all vanish. The noise ε has mean 0 and is independent of the training set, so every term containing ε averages to 0. And E[m − f̂] = m − m = 0, so the term 2(f − m)·E[m − f̂] is 0 as well.
- Name what is left: (f − m)² is the bias squared, E[(f̂ − m)²] is the variance of the model, and σ² is the noise.
E[(y − f̂)²] = (f − E[f̂])² + E[(f̂ − E[f̂])²] + σ² = bias² + variance + noise. The noise term does not depend on the model at all.
Part of Unit 2: Classical ML, built by you in the free ML Foundations course.