Cross-validation and data leakage
Cross-validation scores a model by rotating which slice of the data is held out, so every row is tested once. Data leakage is any way that information from the held-out data sneaks into training.
Why it matters
One train/validation split can be lucky or unlucky, especially with small datasets, so a single score can mislead you by several percent. Cross-validation gives a steadier score and tells you how much it wobbles. Leakage is the most common reason a model that looked brilliant in testing fails in real use, so you need to know where it hides.
The idea
Cross-validation splits the data into k parts: train on k − 1 parts, test on the last one, and rotate. Data leakage is when information from the test data sneaks into training, so results look better than they really are.
Leakage is like a student seeing the exam answers the night before. The high score is real, but it means nothing.
Cheating is on purpose. Leakage is usually an accident, hidden in preprocessing, so you must hunt for it.
How it works
- Shuffle the rows and cut them into k equal folds, for example k = 5.
- Round 1: train on folds 2–5, score on fold 1. Round 2: train on folds 1, 3, 4 and 5, score on fold 2. And so on.
- After k rounds every row has been scored exactly once, by a model that never saw it.
- Report the mean of the k scores and their spread (standard deviation). A big spread means one split could have misled you.
- Every preprocessing step that learns from data (scaling, filling missing values, picking features) must be fitted inside each round, on that round's training folds only.
- Group related rows: all scans of one patient must sit in the same fold. For time series, always train on earlier data and test on later data.
Key points
- k-fold: k models, k scores; report mean ± spread.
- k = 5 or 10 is typical; leave-one-out (k = n) is costly and noisy.
- Stratified folds keep the class balance the same in every fold.
- Group k-fold keeps related rows together; time-series splits only look forwards.
- Leakage: anything at training time that would not be known at prediction time.
- Suspiciously good results are a reason to hunt for leakage, not to celebrate.
Common mistakes
- Scaling or selecting features on the full dataset before splitting.
- The same person, house or document in both training and test folds.
- A feature computed from the label, or from the future (a "days until churn" column).
- Using cross-validation to choose settings and then reporting that same score as final.
Not true: It is safe to scale the features on the whole dataset before splitting it into folds.
In fact: The scaler then learns the mean and spread of the test rows too, which is leakage. Fit it inside each fold, on the training part only.
Not true: The cross-validation score used to choose the settings is also a fair final score.
In fact: Choosing settings by that score fits them to those folds, so the score is too optimistic. Report the final score on data that was never used for any choice.
Part of Unit 2: Classical ML, built by you in the free ML Foundations course.