Testing numerical code (np.allclose, pytest)

A test is a small program that runs your code on a case where you already know the answer, and fails loudly when the result is wrong. Decimals are compared with a small tolerance.

Why it matters

ML code fails quietly. A wrong axis, a sign error or a broadcasting slip still runs and still prints a loss. It just learns worse, and you may never notice. Tests catch these bugs the moment you make them, when they are cheap to fix. And because computers round decimals, the tests have to allow tiny differences.

The idea

A test runs your code on a case where you already know the answer. With decimals, compare with a small tolerance, because computers round numbers.

Checking a calculator with 2 + 2 before you trust it with your taxes.

Passing easy tests does not prove the code is right. Also test edge cases: zeros, one sample, very large numbers.

How it works

  1. Pick cases where you know the answer: softmax of [0, 0] must be [0.5, 0.5], and any softmax output must sum to 1.
  2. Write a function whose name starts with test_, in a file whose name starts with test_, such as test_softmax.py.
  3. Inside, run your code and check the result with assert. For decimals, use np.allclose(result, expected), or np.testing.assert_allclose, which also prints the values that differed.
  4. Run pytest in the terminal. It finds every test, runs them all, and lists any that fail.
  5. Add edge cases: very large inputs such as [1000, 1001, 1002], negative numbers, a batch with shape (batch, classes), a single element.
  6. When you find a bug, first write a test that fails because of it, then fix the code. The test stays, so the bug can never come back unnoticed.

Key points

Common mistakes

Not true: A good way to get a test's expected value is to compute it with the function you are testing.
In fact: Then the test repeats any bug the function has, and still passes. Work the expected value out by hand, or in a different way.

Not true: What to test: the typical case; edge cases such as zeros or empty input can be left out.
In fact: Bugs live in the edge cases: zeros, huge values, a single sample, empty input. Test those too.

Part of Unit 0: Your toolkit in the free ML Foundations course.