Dot product, projection and orthogonality

The dot product multiplies two vectors entry by entry and adds the results. It measures how much they point the same way, and it gives the length of one vector's shadow on another.

Why it matters

ML constantly asks "how similar are these two things?": two documents in search, a query and a key in attention, an input and a neuron's weights. The dot product answers with one cheap number. Projection, built from the dot product, keeps only the part of a vector that lies along a direction, which is the idea behind least squares and PCA.

The idea

The dot product multiplies matching entries and adds them up. It measures how much two vectors point the same way. A dot product of 0 means they are at 90° (orthogonal). A projection is the 'shadow' of one vector on another.

Pulling a box with a rope at an angle. Only the part of your pull along the floor moves the box. That part is the projection.

Your pull is a physical force in 2D or 3D. A dot product works for any vectors, even word embeddings with 768 numbers.

How it works

  1. Multiply matching entries and add: for a = [3, 4] and b = [1, 0], a·b = 3·1 + 4·0 = 3.
  2. The same number in geometry: a·b = |a|·|b|·cos θ, where θ is the angle between them. |a| = 5 and |b| = 1, so cos θ = 3/5 and θ ≈ 53°.
  3. The sign gives the direction: positive means the angle is under 90°, zero means exactly 90° (orthogonal), negative means over 90°.
  4. The projection of a onto b is a's shadow on the line through b: (a·b / b·b)·b = (3/1)·[1, 0] = [3, 0].
  5. What is left over, a − projection = [0, 4], is orthogonal to b: [0, 4]·[1, 0] = 0. Projection splits a vector into "along b" plus "at right angles to b".
  6. Cosine similarity divides out the lengths: cos θ = a·b / (|a||b|). It runs from −1 (opposite) to 1 (same direction) and ignores length, which is why search uses it.

Key points

Common mistakes

Not true: Similarity of two vectors: the raw dot product a·b, whether they are long or short.
In fact: A long vector gets a large dot product just for being long. Cosine similarity divides by both lengths, so only the direction counts.

Not true: Cosine similarity with a zero-length vector: 0.
In fact: A zero-length vector makes the formula divide by 0, so the cosine has no value at all. Check for zero vectors before dividing.

Part of Unit 1: Math foundations in the free ML Foundations course.