Matrices as transformations; matrix multiplication
A matrix is a recipe for moving every vector in space at once; multiplying two matrices means doing one movement after the other.
Why it matters
A neural network layer, a rotation of an image, a change of units, squeezing data down to fewer features: all of these are "do the same linear thing to every vector". A matrix stores that movement in a small grid of numbers, and matrix multiplication lets a computer apply it to millions of vectors, or chain several movements, very fast.
The idea
A matrix is a machine that moves every vector in space. Its columns show where the basis vectors land. Multiplying two matrices means doing one movement after the other.
Instructions for a photo editor: rotate, then stretch.
In the analogy the order feels unimportant, but it matters. Rotate-then-stretch is usually different from stretch-then-rotate, so AB ≠ BA.
How it works
- Watch where the two basis vectors land. A = [[2, 0], [0, 3]] sends [1, 0] to [2, 0], its first column, and [0, 1] to [0, 3], its second column.
- Every other vector follows, because it is a combination of the basis vectors: [1, 1] = [1, 0] + [0, 1] lands on [2, 0] + [0, 3] = [2, 3].
- That is matrix-times-vector: Av = v₁·(first column) + v₂·(second column). Grid lines stay straight, parallel and evenly spaced, and the origin stays put.
- To do B first and then A, multiply AB, read right to left. Each column of AB is A applied to the matching column of B.
- Order matters. With R a 90° rotation, A·R (rotate, then stretch) is [[0, −2], [3, 0]], but R·A (stretch, then rotate) is [[0, −3], [2, 0]].
- Shapes must fit: an (m, n) matrix times an (n, p) matrix gives an (m, p) matrix. The two inner sizes must match.
Key points
- The columns of a matrix are where the basis vectors land. Read a matrix column by column.
- A linear transformation keeps grid lines straight, parallel and evenly spaced, and keeps the origin fixed.
- The product AB means "do B, then A". In general AB ≠ BA.
- Shapes: (m, n) @ (n, p) → (m, p). The inner sizes must match.
- A neural network layer is y = Wx + b: a matrix movement, then a shift.
- In NumPy, @ is matrix multiplication. * multiplies matching entries, which is something else.
Common mistakes
- Using * instead of @ in NumPy. A * B multiplies entries in the same position; it is not the matrix product.
- Reading AB left to right. B happens first.
- Assuming AB = BA. It is rarely true.
- Mixing up the shape: a (3, 2) matrix takes 2-D vectors to 3-D vectors, not the other way round.
Not true: In NumPy, A * B computes the matrix product of A and B.
In fact: A * B multiplies the entries in the same position. The matrix product is written A @ B.
Not true: For two matrices, AB always equals BA.
In fact: The order matters. AB means do B first, then A, and swapping them usually gives a different result.
Part of Unit 1: Math foundations in the free ML Foundations course.