The ML setup: features, labels, loss, train/validation/test

Supervised learning turns examples, each with features and a label, into a model. You fit it on a training set, make your choices on a validation set, and measure the final result once on a test set.

Why it matters

A model that scores well on the data it learned from may still fail on anything new; it may simply have memorised. To know how it will really behave, you must measure it on data it never saw, and you must not let that data influence any of your choices. The three-way split is the simplest honest way to do that.

The idea

Features are the inputs (house size, number of rooms). The label is the answer (price). The loss measures how wrong the model is. Train on one part of the data, tune on the validation set, and use the test set only once, at the end.

Studying for an exam. Training data is textbook exercises. The validation set is practice exams. The test set is the real exam, seen once.

If you studied the real exam paper beforehand, your score would mean nothing. It is the same when you use the test set for tuning: your result becomes a lie.

How it works

  1. Decide what one example is (a house), its features (size, rooms, age) and its label (price). Every row of X is one example; y holds the labels.
  2. Pick a loss that says how wrong a prediction is: squared error for numbers, cross-entropy for categories.
  3. Shuffle the rows, then split them, for example 70% training, 15% validation and 15% test. No row may appear in two sets.
  4. Fit the model's weights on the training set only.
  5. Use the validation set to compare models and settings (learning rate, tree depth, k) and keep the best one.
  6. When every choice is final, score the chosen model once on the test set. That number is your honest estimate for new data.
  7. If you go back and change things after looking at the test score, the test set has become a second validation set, and you need fresh test data.

Key points

Common mistakes

Not true: Several photos of the same person can safely be spread across the training set and the test set.
In fact: The model is then tested on a face it has already seen, so the score is too good. Keep rows that belong together on the same side of the split.

Not true: Tuning a model on the test set just a little does no harm to the final score.
In fact: Every choice made by looking at the test set fits the model to it, so the final score over-promises. Make choices on the validation set, and use the test set once.

Part of Unit 2: Classical ML, built by you in the free ML Foundations course.