Broadcasting and vectorisation

Vectorisation means doing maths on whole arrays in one operation; broadcasting is NumPy's rule for combining arrays of different shapes by stretching any direction of size 1.

Why it matters

Python loops are slow, because the interpreter handles one number at a time. ML needs the same operation on millions of numbers, such as adding a bias to every row of a batch. Vectorised NumPy runs the whole operation in compiled code instead, and broadcasting saves you from building big copies by hand just to make shapes match.

The idea

Vectorisation means one operation works on a whole array at once, instead of a Python loop. Broadcasting lets arrays with different shapes work together, by stretching dimensions of size 1.

A teacher says once: 'everyone add 5 to your score'. She does not walk to each student.

A teacher can talk to any group. Broadcasting only works when the shapes match from the right, or one of them is 1. Otherwise you get an error, or worse, a silent wrong result.

How it works

  1. Write the two shapes one above the other, lined up on the right: (3, 1) and (1, 3).
  2. Compare each pair of sizes, starting from the right. A pair fits if the sizes are equal or one of them is 1. Here 1 fits 3, and 3 fits 1.
  3. If one array has fewer axes, its shape is padded with 1s on the left: (3,) is treated as (1, 3). That is why a 1-D array acts like a row.
  4. Each size-1 direction is stretched to match the other array. scores is stretched across to 3 columns and bonus is stretched down to 3 rows. Nothing is really copied in memory.
  5. The operation then runs cell by cell: result[i, j] = scores[i, 0] + bonus[0, j]. The result takes the larger size in each position, so it is (3, 3).
  6. If any pair is unequal and neither is 1, such as (3,) and (4,), NumPy stops with "operands could not be broadcast together".

Key points

Common mistakes

Not true: Shapes (n,) + (n, 1): NumPy raises an error.
In fact: It raises no error: broadcasting quietly produces an (n, n) grid. That is why you should assert the shapes you expect.

Not true: Adding a row to every row of a matrix: copy the row first with np.tile.
In fact: Broadcasting adds the row to every row without copying anything. np.tile would only use up memory.

Part of Unit 0: Your toolkit in the free ML Foundations course.