Eigenvalues and eigenvectors

An eigenvector of a matrix is a direction the matrix does not turn, only stretches. Its eigenvalue is how much it is stretched.

Why it matters

A matrix moves vectors in a confusing mix of turning and stretching. Eigenvectors find the special directions where it acts like plain multiplication, which makes the matrix easy to understand. They give the main directions of variation in data (PCA), tell you whether repeated multiplication explodes or fades (RNNs, Markov chains), and explain how fast gradient descent converges.

The idea

An eigenvector is a direction that a matrix does not turn. It only stretches or shrinks it. The eigenvalue is the stretch factor: Av = λv.

A spinning globe. Every point moves, except the points on the axis. The axis is an eigenvector, with λ = 1.

A 2D rotation has no real eigenvectors at all: it turns every direction.

How it works

  1. Apply A = [[2, 1], [1, 2]] to many directions and see which ones stay on their own line. Most are knocked off: [1, 0] becomes [2, 1], a different direction.
  2. [1, 1] becomes [3, 3] = 3·[1, 1]: same direction, 3 times longer. That is an eigenvector with eigenvalue 3.
  3. [1, −1] becomes [1, −1]: unchanged. An eigenvector with eigenvalue 1.
  4. To find them by hand, solve det(A − λI) = 0. Here (2 − λ)² − 1 = 0, so λ = 3 or λ = 1.
  5. For each λ, solve (A − λI)v = 0 to get its direction. Any non-zero multiple of v is also an eigenvector.
  6. In the eigenvector directions, A just stretches: 3× along [1, 1] and 1× along [1, −1]. Applying A ten times stretches by 3¹⁰ and 1¹⁰, so repeated use is easy to predict.

Key points

Common mistakes

Not true: Every matrix has at least one real direction that it only stretches.
In fact: A rotation in 2-D turns every direction, so it has no real eigenvector: its eigenvalues are complex numbers.

Not true: An eigenvector has length 1, and a longer copy of it is not an eigenvector.
In fact: Any multiple of an eigenvector is an eigenvector too. Libraries return length 1 only as a convention.

Go deeper: Why multiplying by A again and again finds the biggest eigenvector

  1. Take a matrix with eigenvectors v₁ and v₂ and eigenvalues λ₁ and λ₂, where |λ₁| is the bigger one. In the lesson's example, v₁ = [1, 1] with λ₁ = 3, and v₂ = [1, −1] with λ₂ = 1.
  2. Any starting vector is a mix of the two: x = c₁v₁ + c₂v₂. For example [1, 0] = 0.5·[1, 1] + 0.5·[1, −1].
  3. Apply A once. Each eigenvector is only stretched, so Ax = c₁λ₁v₁ + c₂λ₂v₂.
  4. Apply it k times: Aᵏx = c₁λ₁ᵏv₁ + c₂λ₂ᵏv₂ = λ₁ᵏ·(c₁v₁ + c₂(λ₂/λ₁)ᵏv₂).
  5. Because |λ₂/λ₁| < 1, the second part shrinks every round: here by a factor of 3 each time. After 10 rounds it is 59,049 times smaller.
  6. Rescale the vector to length 1 after each round, so the numbers don't overflow. What is left points along v₁. This method is called power iteration.

Aᵏx lines up with the eigenvector of the largest |λ|, and the error shrinks by |λ₂/λ₁| each round. It needs c₁ ≠ 0 and one eigenvalue that is strictly the largest.

Part of Unit 1: Math foundations in the free ML Foundations course.