Broadcasting and vectorisation
Vectorisation means doing maths on whole arrays in one operation; broadcasting is NumPy's rule for combining arrays of different shapes by stretching any direction of size 1.
Why it matters
Python loops are slow, because the interpreter handles one number at a time. ML needs the same operation on millions of numbers, such as adding a bias to every row of a batch. Vectorised NumPy runs the whole operation in compiled code instead, and broadcasting saves you from building big copies by hand just to make shapes match.
The idea
Vectorisation means one operation works on a whole array at once, instead of a Python loop. Broadcasting lets arrays with different shapes work together, by stretching dimensions of size 1.
A teacher says once: 'everyone add 5 to your score'. She does not walk to each student.
A teacher can talk to any group. Broadcasting only works when the shapes match from the right, or one of them is 1. Otherwise you get an error, or worse, a silent wrong result.
How it works
- Write the two shapes one above the other, lined up on the right: (3, 1) and (1, 3).
- Compare each pair of sizes, starting from the right. A pair fits if the sizes are equal or one of them is 1. Here 1 fits 3, and 3 fits 1.
- If one array has fewer axes, its shape is padded with 1s on the left: (3,) is treated as (1, 3). That is why a 1-D array acts like a row.
- Each size-1 direction is stretched to match the other array. scores is stretched across to 3 columns and bonus is stretched down to 3 rows. Nothing is really copied in memory.
- The operation then runs cell by cell: result[i, j] = scores[i, 0] + bonus[0, j]. The result takes the larger size in each position, so it is (3, 3).
- If any pair is unequal and neither is 1, such as (3,) and (4,), NumPy stops with "operands could not be broadcast together".
Key points
- The rule: line the shapes up from the right; each pair must be equal or contain a 1.
- Missing leading axes count as 1, so (3,) behaves like (1, 3).
- Vectorised code is usually 10 to 100 times faster than a Python loop, and shorter.
- x[:, None] (or x.reshape(-1, 1)) turns a shape (n,) array into a column (n, 1); x[None, :] turns it into a row (1, n).
- Everyday ML uses: X - X.mean(axis=0) subtracts each feature's mean from every row; adding a bias to every row of a batch; all pairwise distances in one line.
- keepdims=True keeps the summed axis with size 1, so the result broadcasts back: X / X.sum(axis=1, keepdims=True) makes every row sum to 1.
Common mistakes
- The silent bug: adding shapes (n,) and (n, 1) gives an (n, n) result and no error. Assert the shapes you expect.
- Forgetting keepdims=True. X.sum(axis=1) has shape (n,), which lines up with the columns, not the rows: an error, or for a square X, a silently wrong answer.
- Copying data with np.tile or np.repeat just to make shapes match. Broadcasting does this without using the memory.
- Writing a Python loop over rows for something one array expression can do.
Not true: Shapes (n,) + (n, 1): NumPy raises an error.
In fact: It raises no error: broadcasting quietly produces an (n, n) grid. That is why you should assert the shapes you expect.
Not true: Adding a row to every row of a matrix: copy the row first with np.tile.
In fact: Broadcasting adds the row to every row without copying anything. np.tile would only use up memory.
Part of Unit 0: Your toolkit in the free ML Foundations course.