The ML setup: features, labels, loss, train/validation/test
Supervised learning turns examples, each with features and a label, into a model. You fit it on a training set, make your choices on a validation set, and measure the final result once on a test set.
Why it matters
A model that scores well on the data it learned from may still fail on anything new; it may simply have memorised. To know how it will really behave, you must measure it on data it never saw, and you must not let that data influence any of your choices. The three-way split is the simplest honest way to do that.
The idea
Features are the inputs (house size, number of rooms). The label is the answer (price). The loss measures how wrong the model is. Train on one part of the data, tune on the validation set, and use the test set only once, at the end.
Studying for an exam. Training data is textbook exercises. The validation set is practice exams. The test set is the real exam, seen once.
If you studied the real exam paper beforehand, your score would mean nothing. It is the same when you use the test set for tuning: your result becomes a lie.
How it works
- Decide what one example is (a house), its features (size, rooms, age) and its label (price). Every row of X is one example; y holds the labels.
- Pick a loss that says how wrong a prediction is: squared error for numbers, cross-entropy for categories.
- Shuffle the rows, then split them, for example 70% training, 15% validation and 15% test. No row may appear in two sets.
- Fit the model's weights on the training set only.
- Use the validation set to compare models and settings (learning rate, tree depth, k) and keep the best one.
- When every choice is final, score the chosen model once on the test set. That number is your honest estimate for new data.
- If you go back and change things after looking at the test score, the test set has become a second validation set, and you need fresh test data.
Key points
- Feature = input; label = the answer; example = one row of both.
- Training set: learn weights. Validation set: make choices. Test set: report the final score, once.
- Shuffle before splitting, unless the data has an order that matters (time series: train on the past, test on the future).
- With little data, use cross-validation instead of a single validation set.
- Keep the class balance similar in every split (a "stratified" split) when one class is rare.
- Always compare against a simple baseline, such as predicting the average or the most common class.
Common mistakes
- Tuning on the test set, even "just a little". The final score then over-promises.
- Splitting rows that belong together (several photos of the same person) across train and test.
- Shuffling time-series data, so the model trains on the future.
- Reporting training accuracy as if it were real-world accuracy.
Not true: Several photos of the same person can safely be spread across the training set and the test set.
In fact: The model is then tested on a face it has already seen, so the score is too good. Keep rows that belong together on the same side of the split.
Not true: Tuning a model on the test set just a little does no harm to the final score.
In fact: Every choice made by looking at the test set fits the model to it, so the final score over-promises. Make choices on the validation set, and use the test set once.
Part of Unit 2: Classical ML, built by you in the free ML Foundations course.