Inverse, rank and solving Ax = b
The inverse of a matrix undoes what the matrix does. The rank counts how many independent directions survive it, and when the rank is full, Ax = b has exactly one solution.
Why it matters
Many ML problems end in "solve these equations": the best weights of linear regression satisfy XᵀX w = Xᵀy. You need to know when there is exactly one answer, when there is none or infinitely many, and how to compute it safely. Rank and the inverse answer exactly those questions.
The idea
The inverse undoes a matrix. Rank is the number of independent directions that survive the matrix. If the rank is full, Ax = b has exactly one solution, x = A⁻¹b.
The inverse is an undo button. Rank is like taking a photo of a 3D object: the photo keeps only 2 of the 3 directions, so its rank is 2.
In an editor, undo always works. If a matrix squashes space (rank too low), information is lost, and there is no undo.
How it works
- Write the equations 2x + y = 3 and x + 3y = 5 as a matrix: A = [[2, 1], [1, 3]] and b = [3, 5]. The question becomes: which vector does A move onto b?
- Check the rank first. The two columns point in different directions, so the rank is 2 (full), and det = 2·3 − 1·1 = 5, not 0.
- Full rank means A can be undone. For 2 × 2, the inverse is (1/det)·[[d, −b], [−c, a]] = (1/5)·[[3, −1], [−1, 2]].
- Apply the inverse to b: x = A⁻¹b = (1/5)·[3·3 − 5, −3 + 2·5] = [0.8, 1.4].
- Check by multiplying back: 2(0.8) + 1.4 = 3 and 0.8 + 3(1.4) = 5.
- If the rank is not full, as with [[1, 2], [2, 4]] (rank 1), A squashes the plane onto a line. Then b is either off that line (no solution) or on it (infinitely many solutions).
- In code, call np.linalg.solve(A, b). It uses elimination, which is faster and more accurate than computing inv(A) and multiplying.
Key points
- Rank = the number of independent columns = the dimension of everything A can reach (its column space).
- Full-rank square matrix: invertible, det ≠ 0, exactly one solution for every b.
- Rank too low: some direction is squashed to zero (the null space). Information is lost, and there is no inverse.
- A⁻¹A = I, the identity matrix, which changes nothing.
- Use np.linalg.solve, not np.linalg.inv(A) @ b.
- With more equations than unknowns (a tall data matrix), there is usually no exact solution. Least squares finds the closest one: that is linear regression.
Common mistakes
- Computing inv(A) @ b. It is slower and less accurate, especially when A is nearly singular.
- Assuming every square matrix has an inverse. Check the rank or the condition number.
- Mixing up "no solution" and "infinitely many": both come from too low a rank, depending on b.
Not true: The most accurate way to solve Ax = b in code is to compute the inverse of A and multiply it by b.
In fact: Building the inverse is slower and less accurate, especially when A is nearly singular. np.linalg.solve finds x without ever building it.
Not true: Every square matrix has an inverse.
In fact: A square matrix that squashes space flat (its rank is too low, its determinant is 0) has no inverse. Check the rank or the condition number first.
Part of Unit 1: Math foundations in the free ML Foundations course.