PCA in practice
In practice, PCA is used to compress features, remove noise, speed up other models and plot high-dimensional data, keeping just enough components to hold most of the variance.
Why it matters
You now know how PCA works; the hard part is using it well. How many components should you keep? Should you scale first? When does it help a downstream model, and when does it throw away the very signal you need? Those practical choices decide whether PCA helps or hurts.
The idea
Use PCA to compress data, remove noise, or plot high-dimensional data in 2D. Always centre the data first, and scale it if features use different units.
Photographing a 3D object from the angle that shows the most detail.
The angle with the most spread is not always the most useful one. PCA ignores the labels, so the direction that separates your classes might be dropped.
How it works
- Fit on training data only: compute the mean (and, if units differ, the standard deviation) from the training rows.
- Centre (and scale) the data, then run SVD.
- Plot the explained-variance ratio per component (a scree plot) and its running total.
- Pick k where the running total reaches your target, often 90–99%, or at the elbow of the scree plot.
- Transform: keep the top-k component scores as the new features. Apply the same transform to validation and test data.
- For visualisation, plot the first two or three components, coloured by label.
- To check what was lost, reconstruct the data from k components and compare it with the original.
Key points
- Centre always; standardise when features have different units.
- Choose the number of components k from the cumulative explained variance: keep enough to reach, say, 95%.
- PCA ignores labels: maximum spread is not always maximum usefulness.
- It removes correlated, redundant features and can denoise data.
- Non-linear structure needs other tools (t-SNE or UMAP for plotting, autoencoders in Unit 7).
- For MNIST digits, about 150 of 784 components keep around 95% of the variance.
Common mistakes
- Fitting PCA on the full dataset, test set included.
- Not scaling features that have different units: the one with the largest numbers becomes the first component.
- Assuming components have a human meaning.
- Dropping low-variance components that actually separate the classes.
Not true: Each principal component stands for one real-world quantity, such as income or age.
In fact: A component is a mix of all the original features, chosen because the data spreads out along it. It usually has no simple meaning.
Not true: Low-variance components: they carry no information that separates the classes.
In fact: PCA ignores the labels. A direction with little spread can be exactly the one that separates the classes, so check before dropping it.
Part of Unit 2: Classical ML, built by you in the free ML Foundations course.