Dot product, projection and orthogonality
The dot product multiplies two vectors entry by entry and adds the results. It measures how much they point the same way, and it gives the length of one vector's shadow on another.
Why it matters
ML constantly asks "how similar are these two things?": two documents in search, a query and a key in attention, an input and a neuron's weights. The dot product answers with one cheap number. Projection, built from the dot product, keeps only the part of a vector that lies along a direction, which is the idea behind least squares and PCA.
The idea
The dot product multiplies matching entries and adds them up. It measures how much two vectors point the same way. A dot product of 0 means they are at 90° (orthogonal). A projection is the 'shadow' of one vector on another.
Pulling a box with a rope at an angle. Only the part of your pull along the floor moves the box. That part is the projection.
Your pull is a physical force in 2D or 3D. A dot product works for any vectors, even word embeddings with 768 numbers.
How it works
- Multiply matching entries and add: for a = [3, 4] and b = [1, 0], a·b = 3·1 + 4·0 = 3.
- The same number in geometry: a·b = |a|·|b|·cos θ, where θ is the angle between them. |a| = 5 and |b| = 1, so cos θ = 3/5 and θ ≈ 53°.
- The sign gives the direction: positive means the angle is under 90°, zero means exactly 90° (orthogonal), negative means over 90°.
- The projection of a onto b is a's shadow on the line through b: (a·b / b·b)·b = (3/1)·[1, 0] = [3, 0].
- What is left over, a − projection = [0, 4], is orthogonal to b: [0, 4]·[1, 0] = 0. Projection splits a vector into "along b" plus "at right angles to b".
- Cosine similarity divides out the lengths: cos θ = a·b / (|a||b|). It runs from −1 (opposite) to 1 (same direction) and ignores length, which is why search uses it.
Key points
- The dot product two ways: a·b = Σ aᵢbᵢ (multiply matching entries and add) = |a||b| cos θ (the two lengths times the cosine of the angle between them).
- A dot product of 0 means the two vectors are orthogonal: at 90° to each other.
- A vector dotted with itself gives its length squared: a·a = |a|², so its length is √(a·a).
- Projection of a onto b: (a·b / b·b)·b. The leftover part is orthogonal to b.
- Cosine similarity is the dot product after scaling both vectors to length 1.
- A matrix–vector product is a list of dot products: each output is one row of the matrix dotted with the vector.
Common mistakes
- Comparing similarity with the raw dot product when lengths differ a lot. Long vectors win just for being long; use cosine similarity.
- Forgetting to divide by b·b in the projection when b is not length 1.
- Dividing by a zero-length vector in cosine similarity.
Not true: Similarity of two vectors: the raw dot product a·b, whether they are long or short.
In fact: A long vector gets a large dot product just for being long. Cosine similarity divides by both lengths, so only the direction counts.
Not true: Cosine similarity with a zero-length vector: 0.
In fact: A zero-length vector makes the formula divide by 0, so the cosine has no value at all. Check for zero vectors before dividing.
Part of Unit 1: Math foundations in the free ML Foundations course.