SVD, and the maths behind PCA

SVD splits any matrix into three simple moves: rotate, stretch, rotate. PCA uses it to find the directions in which data varies most, so you can keep a few and drop the rest.

Why it matters

Real data has many features, and many move together: height and weight, or neighbouring pixels. Working in fewer dimensions is faster, easier to plot and often less noisy. PCA finds the few directions that keep most of the information, and SVD is the reliable way to compute them for any matrix, square or not.

The idea

SVD splits any matrix into three simple steps: rotate, stretch, rotate (A = UΣVᵀ). The biggest stretch values show the most important directions. PCA keeps only those directions of the data.

A cloud of points shaped like a rugby ball has a long axis, a medium axis and a short axis. PCA keeps the long axes and drops the short ones.

PCA only finds straight directions. If the data lies on a curve, like a spiral, PCA misses the real shape.

How it works

  1. Centre the data: subtract the mean of each feature, so the cloud sits around the origin. PCA measures spread around the mean, so this step is essential.
  2. Run SVD on the centred data matrix X (n samples × d features): X = UΣVᵀ.
  3. The rows of Vᵀ are the principal directions, ordered from most to least spread. The first points along the long axis of the cloud.
  4. The singular values in Σ say how much spread each direction has: the variance along direction k is σₖ² / (n − 1).
  5. The share of variance explained by direction k is σₖ² / Σσ². If the first two directions explain 95%, two numbers per sample keep almost everything.
  6. Project onto the top k directions: Z = X·Vₖ. Each sample is now k numbers instead of d.
  7. To go back approximately, compute Z·Vₖᵀ and add the mean. What is lost is exactly the spread along the dropped directions.

Key points

Common mistakes

Not true: PCA finds the same directions whether or not you subtract the mean first.
In fact: PCA measures spread around the mean. Without centring, the first direction mostly points from the origin towards the mean of the data.

Not true: Principal directions = the causes behind the data.
In fact: They are only the directions along which the data spreads out most. PCA says nothing about what causes what.

Go deeper: Why the direction of most variance is an eigenvector

  1. Centre the data and call its covariance matrix C = XᵀX / (n − 1). Take any direction u of length 1.
  2. Project every sample onto u: the scores are Xu. Their variance is (Xu)ᵀ(Xu) / (n − 1) = uᵀCu.
  3. PCA asks for the u that makes uᵀCu as large as possible, under the constraint uᵀu = 1. Without the constraint you could simply make u longer.
  4. Use a Lagrange multiplier (the last lesson of this unit): L = uᵀCu − λ(uᵀu − 1). Setting the gradient with respect to u to zero gives 2Cu − 2λu = 0, so Cu = λu.
  5. So the best u has to be an eigenvector of C. Which one? Put Cu = λu back into the variance: uᵀCu = λ·uᵀu = λ.
  6. The variance along an eigenvector equals its eigenvalue, so the largest eigenvalue wins. Ask again for the best direction at 90° to the ones already chosen, and you get the second eigenvector, then the third.

The principal directions are the eigenvectors of the covariance matrix, and the variance along each one is its eigenvalue, λₖ = σₖ² / (n − 1).

Part of Unit 1: Math foundations in the free ML Foundations course.