Derivatives and the chain rule

A derivative is the rate at which an output changes when you nudge the input; the chain rule says that for a function inside a function, you multiply the rates.

Why it matters

Training needs to know, for every weight: "if I nudge this a little, does the loss go up or down, and how fast?" That is a derivative. A network is functions nested inside functions, many layers deep, so the chain rule is what turns one loss into a derivative for every weight. Backpropagation is the chain rule, organised efficiently.

The idea

A derivative is a rate of change: how much the output moves when the input moves a tiny bit. The chain rule: for a function inside a function, multiply the rates.

Gears. If gear A turns gear B 3 times faster, and B turns C 2 times faster, then A turns C 3 × 2 = 6 times faster.

Gear ratios are fixed. Derivatives change from point to point, so you calculate them at the current value.

How it works

  1. Nudge the input and watch the output. For f(x) = (3x + 1)²: at x = 1, f = 16; at x = 1.01, f = 16.2409. The output moved 0.2409 for a 0.01 nudge, a rate of about 24.
  2. The derivative is that rate as the nudge shrinks towards zero. On a graph, it is the slope of the tangent line at that point.
  3. Split the function into a chain: u = 3x + 1 on the inside, then f = u² on the outside.
  4. Find the rate of each link: du/dx = 3 and df/du = 2u. At x = 1, u = 4, so df/du = 8.
  5. Multiply the rates along the chain: df/dx = df/du · du/dx = 8 · 3 = 24. A 0.01 nudge in x moves u by 0.03, which moves f by 0.24.
  6. Check numerically: (f(x + h) − f(x − h)) / 2h with h = 1e-5 gives 24.0000. Always check hand-derived gradients this way.

Key points

Common mistakes

Not true: Numerical derivative: the smaller the step h, the more accurate the answer.
In fact: Only up to a point. With h as small as 1e-15, rounding errors are bigger than the tiny difference, and the answer becomes wrong.

Not true: d/dx (3x + 1)² = 2(3x + 1).
In fact: The chain rule also multiplies by the inner rate, which is 3. The derivative is 2(3x + 1) × 3 = 6(3x + 1).

Part of Unit 1: Math foundations in the free ML Foundations course.