Vectors, span and basis

A vector is a list of numbers that you can draw as an arrow. The span of some vectors is every point you can reach by scaling and adding them, and a basis is the smallest set that reaches the whole space.

Why it matters

Every piece of data in ML becomes a vector: a house is [size, rooms, age], a word is a list of 768 numbers, an image is a long list of pixel values. To reason about data you need to know what you can build from a few vectors, and whether a feature carries new information or just repeats what you already have.

The idea

A vector is an arrow, or a list of numbers. The span is every point you can reach by scaling and adding your vectors. A basis is the smallest set of vectors that can reach every point in the space.

Two ingredients, flour and water. Every mix you can make is the span. A third ingredient that is only 'flour plus water' adds nothing new.

Ingredient amounts cannot be negative. Vector weights can be negative.

How it works

  1. A vector [3, 2] means "3 steps along x, then 2 steps along y". Drawn from the origin it is an arrow; written down it is just two numbers.
  2. Scaling multiplies every entry: 3·[1, 0] = [3, 0]. A negative number flips the arrow round.
  3. Adding goes tip to tail, entry by entry: [3, 0] + [0, 2] = [3, 2]. A sum of scaled vectors is called a linear combination.
  4. The span of some vectors is every linear combination of them. [1, 0] and [0, 1] can reach any point in 2-D, so their span is the whole plane.
  5. Vectors are linearly independent if none of them can be made from the others. [1, 2] and [2, 4] are not: [2, 4] = 2·[1, 2], so together they only span one line.
  6. A basis is a set of independent vectors that spans the space. Every basis of 2-D has exactly 2 vectors; that number is the dimension of the space.
  7. In code, stack the vectors as the columns of a matrix and ask for its rank with np.linalg.matrix_rank. If the rank equals the number of vectors, they are independent.

Key points

Common mistakes

Not true: Adding one more vector to a set always makes its span bigger.
In fact: Only an independent vector adds something. A vector that is a combination of the others, like [2, 4] next to [1, 2], reaches nothing new.

Not true: Span of two vectors: a line or a plane, which may miss the origin.
In fact: Every span contains the origin: scale all the vectors by 0 and you are there.

Part of Unit 1: Math foundations in the free ML Foundations course.