Feature scaling and encoding
Feature scaling puts numeric features on similar ranges, and encoding turns categories into numbers, so that models treat every feature fairly and don't invent orders that aren't there.
Why it matters
Models only see numbers. A feature measured in thousands outweighs one measured in single digits, in any distance or gradient. And coding "red, green, blue" as 1, 2, 3 tells the model that green sits between the other two. Getting this wrong quietly ruins k-NN, SVMs, regularised models and neural networks, while trees barely notice.
The idea
Put features on similar scales so no feature dominates just because of its units. Turn categories into numbers with one-hot encoding.
Describing a house by size (100 m²) and rooms (3). Without scaling, the model 'hears' size shouting and rooms whispering.
Tree models, like decision trees and random forests, do not care about scale. Scaling matters most for distance-based and gradient-based methods.
How it works
- Standardisation: for each numeric feature, subtract the training mean and divide by the training standard deviation. The result has mean 0 and spread 1.
- Min–max scaling squeezes values into 0 to 1 instead. It is handy for pixel values, but sensitive to outliers.
- Fit the scaler on the training rows only, then apply the same mean and standard deviation to validation and test rows.
- After scaling, the loss surface is rounder, so gradient descent heads straight for the minimum instead of zig-zagging.
- One-hot encoding: give each category its own 0/1 column. red = [1, 0, 0], green = [0, 1, 0], blue = [0, 0, 1].
- Plan for categories you have never seen: map them to an "unknown" column, or all zeros.
- For categories with a real order (small < medium < large), ordinal numbers 0, 1, 2 are fine.
Key points
- Distance-based and gradient-based models need scaling. Trees and forests don't.
- Standardise: (x − mean) / std, with statistics from the training rows only.
- One-hot for categories with no order; ordinal numbers only for real orders.
- Categories with thousands of values (postcodes, user ids) need other tricks: hashing, frequency or learned embeddings (Unit 3).
- Log-transform heavily skewed values like prices or incomes before scaling.
- Keep the fitted scaler and encoder with the model: they are part of it.
Common mistakes
- Fitting the scaler on all the data: leakage.
- Coding unordered categories (red, green, blue) as 1, 2, 3, which tells the model that blue is "more" than red.
- Forgetting to apply the same scaling at prediction time.
- One-hot encoding a feature with 50,000 values into 50,000 columns.
Not true: At prediction time: new data goes into the model as it is, without the scaler.
In fact: The model learned from scaled numbers, so new inputs must go through the same scaler, using the mean and spread of the training data.
Not true: Red, green, blue coded as 1, 2, 3: a neutral choice that tells the model nothing false.
In fact: The numbers tell the model that blue is more than red, and that green lies between them. For categories with no order, use one-hot encoding.
Part of Unit 2: Classical ML, built by you in the free ML Foundations course.