Ensembles: bagging, random forests, gradient boosting
An ensemble combines many models. Bagging and random forests average trees trained on random resamples, which cuts variance. Boosting adds small trees one after another, each fixing the errors left so far, which cuts bias.
Why it matters
A single deep tree is unstable: change a few rows and it changes completely. A single shallow tree is too simple. Ensembles get the best of both, and gradient-boosted trees (XGBoost, LightGBM) and random forests are still the strongest models for most tabular data in industry.
The idea
Ensembles combine many models. Bagging and random forests train many trees on random samples of the data and features, then average them (this reduces variance). Boosting trains trees one after another, each one fixing the errors of the ones before (this reduces bias).
Random forest: ask 100 people to guess the number of sweets in a jar, and average the guesses. Boosting: one student corrects their homework again and again, focusing on the questions they got wrong.
Averaging only helps if people make different mistakes. If every tree were the same, the forest would be no better than one tree. That is why forests also pick random features.
How it works
- Bagging: draw a bootstrap sample (n rows chosen at random, with repeats) and grow a deep tree on it. Repeat to get, say, 100 different trees.
- Random forest: also let each split consider only a random subset of the features, so the trees differ even more.
- Predict by averaging the trees (regression) or by majority vote (classification). Their different mistakes cancel out, so variance drops.
- Boosting starts from a simple prediction, such as the average.
- Compute what is still wrong: the residuals y − prediction.
- Fit a small tree to those residuals and add a small step of it (the learning rate, often 0.1) to the prediction.
- Repeat hundreds of times. Each tree fixes what the ones before got wrong. Stop when validation error stops improving.
Key points
- Bagging and forests: parallel, deep trees, averaged → lower variance.
- Boosting: sequential, shallow trees, added up → lower bias.
- Averaging only helps if the models make different mistakes, which is why forests also randomise features.
- Forests are hard to overfit by adding trees; boosting can overfit if it runs too long or with too high a learning rate.
- Out-of-bag rows (the ones left out of each bootstrap sample) give a free validation score for forests.
- Gradient boosting is gradient descent in "function space": each tree is a step along the negative gradient of the loss.
Common mistakes
- Using identical trees: no randomness means no benefit from averaging.
- Boosting with a large learning rate and many rounds, without early stopping.
- Expecting an ensemble to be as easy to explain as one tree.
Not true: Gradient boosting can run for as many rounds as you like without overfitting.
In fact: Each round fits the errors that remain, noise included. With many rounds and a large learning rate it overfits, so stop early using validation data.
Not true: Averaging many identical trees gives a better model than any one of them.
In fact: Identical trees make identical mistakes, so their average is the same tree. Averaging helps only when the trees differ, which is why forests add randomness.
Part of Unit 2: Classical ML, built by you in the free ML Foundations course.