Ensembles: bagging, random forests, gradient boosting

An ensemble combines many models. Bagging and random forests average trees trained on random resamples, which cuts variance. Boosting adds small trees one after another, each fixing the errors left so far, which cuts bias.

Why it matters

A single deep tree is unstable: change a few rows and it changes completely. A single shallow tree is too simple. Ensembles get the best of both, and gradient-boosted trees (XGBoost, LightGBM) and random forests are still the strongest models for most tabular data in industry.

The idea

Ensembles combine many models. Bagging and random forests train many trees on random samples of the data and features, then average them (this reduces variance). Boosting trains trees one after another, each one fixing the errors of the ones before (this reduces bias).

Random forest: ask 100 people to guess the number of sweets in a jar, and average the guesses. Boosting: one student corrects their homework again and again, focusing on the questions they got wrong.

Averaging only helps if people make different mistakes. If every tree were the same, the forest would be no better than one tree. That is why forests also pick random features.

How it works

  1. Bagging: draw a bootstrap sample (n rows chosen at random, with repeats) and grow a deep tree on it. Repeat to get, say, 100 different trees.
  2. Random forest: also let each split consider only a random subset of the features, so the trees differ even more.
  3. Predict by averaging the trees (regression) or by majority vote (classification). Their different mistakes cancel out, so variance drops.
  4. Boosting starts from a simple prediction, such as the average.
  5. Compute what is still wrong: the residuals y − prediction.
  6. Fit a small tree to those residuals and add a small step of it (the learning rate, often 0.1) to the prediction.
  7. Repeat hundreds of times. Each tree fixes what the ones before got wrong. Stop when validation error stops improving.

Key points

Common mistakes

Not true: Gradient boosting can run for as many rounds as you like without overfitting.
In fact: Each round fits the errors that remain, noise included. With many rounds and a large learning rate it overfits, so stop early using validation data.

Not true: Averaging many identical trees gives a better model than any one of them.
In fact: Identical trees make identical mistakes, so their average is the same tree. Averaging helps only when the trees differ, which is why forests add randomness.

Part of Unit 2: Classical ML, built by you in the free ML Foundations course.