Vectors, span and basis
A vector is a list of numbers that you can draw as an arrow. The span of some vectors is every point you can reach by scaling and adding them, and a basis is the smallest set that reaches the whole space.
Why it matters
Every piece of data in ML becomes a vector: a house is [size, rooms, age], a word is a list of 768 numbers, an image is a long list of pixel values. To reason about data you need to know what you can build from a few vectors, and whether a feature carries new information or just repeats what you already have.
The idea
A vector is an arrow, or a list of numbers. The span is every point you can reach by scaling and adding your vectors. A basis is the smallest set of vectors that can reach every point in the space.
Two ingredients, flour and water. Every mix you can make is the span. A third ingredient that is only 'flour plus water' adds nothing new.
Ingredient amounts cannot be negative. Vector weights can be negative.
How it works
- A vector [3, 2] means "3 steps along x, then 2 steps along y". Drawn from the origin it is an arrow; written down it is just two numbers.
- Scaling multiplies every entry: 3·[1, 0] = [3, 0]. A negative number flips the arrow round.
- Adding goes tip to tail, entry by entry: [3, 0] + [0, 2] = [3, 2]. A sum of scaled vectors is called a linear combination.
- The span of some vectors is every linear combination of them. [1, 0] and [0, 1] can reach any point in 2-D, so their span is the whole plane.
- Vectors are linearly independent if none of them can be made from the others. [1, 2] and [2, 4] are not: [2, 4] = 2·[1, 2], so together they only span one line.
- A basis is a set of independent vectors that spans the space. Every basis of 2-D has exactly 2 vectors; that number is the dimension of the space.
- In code, stack the vectors as the columns of a matrix and ask for its rank with np.linalg.matrix_rank. If the rank equals the number of vectors, they are independent.
Key points
- Two views of the same thing: an arrow (geometry) and a list of numbers (data).
- Linear combination: scale each vector, then add. Almost everything in linear algebra is built from this.
- The span is all the linear combinations. It is always a line, a plane or a higher-dimensional flat space through the origin.
- Independent: no vector is a combination of the others. A dependent vector adds nothing new to the span.
- Basis: independent and spans the space. In n dimensions every basis has exactly n vectors.
- In ML, a duplicated or derived feature (a price in pounds and in dollars) is a dependent direction: no new information, and it can make some models numerically unstable.
Common mistakes
- Thinking more vectors always means a bigger span. Extra dependent vectors add nothing.
- Forgetting that every span passes through the origin: scale everything by 0 and you are there.
- Testing independence with an exact "== 0" check on decimals. Use the rank with a tolerance, or look at the singular values.
Not true: Adding one more vector to a set always makes its span bigger.
In fact: Only an independent vector adds something. A vector that is a combination of the others, like [2, 4] next to [1, 2], reaches nothing new.
Not true: Span of two vectors: a line or a plane, which may miss the origin.
In fact: Every span contains the origin: scale all the vectors by 0 and you are there.
Part of Unit 1: Math foundations in the free ML Foundations course.