Derivatives and the chain rule
A derivative is the rate at which an output changes when you nudge the input; the chain rule says that for a function inside a function, you multiply the rates.
Why it matters
Training needs to know, for every weight: "if I nudge this a little, does the loss go up or down, and how fast?" That is a derivative. A network is functions nested inside functions, many layers deep, so the chain rule is what turns one loss into a derivative for every weight. Backpropagation is the chain rule, organised efficiently.
The idea
A derivative is a rate of change: how much the output moves when the input moves a tiny bit. The chain rule: for a function inside a function, multiply the rates.
Gears. If gear A turns gear B 3 times faster, and B turns C 2 times faster, then A turns C 3 × 2 = 6 times faster.
Gear ratios are fixed. Derivatives change from point to point, so you calculate them at the current value.
How it works
- Nudge the input and watch the output. For f(x) = (3x + 1)²: at x = 1, f = 16; at x = 1.01, f = 16.2409. The output moved 0.2409 for a 0.01 nudge, a rate of about 24.
- The derivative is that rate as the nudge shrinks towards zero. On a graph, it is the slope of the tangent line at that point.
- Split the function into a chain: u = 3x + 1 on the inside, then f = u² on the outside.
- Find the rate of each link: du/dx = 3 and df/du = 2u. At x = 1, u = 4, so df/du = 8.
- Multiply the rates along the chain: df/dx = df/du · du/dx = 8 · 3 = 24. A 0.01 nudge in x moves u by 0.03, which moves f by 0.24.
- Check numerically: (f(x + h) − f(x − h)) / 2h with h = 1e-5 gives 24.0000. Always check hand-derived gradients this way.
Key points
- Rules to know: d/dx xⁿ = n·xⁿ⁻¹; d/dx eˣ = eˣ; d/dx ln x = 1/x; constants multiply through.
- Chain rule: d/dx f(g(x)) = f′(g(x))·g′(x). Multiply the rates along the chain.
- Positive derivative: the output rises when the input rises. Negative: it falls. Zero: flat, where a minimum can be.
- The sigmoid σ(x) = 1/(1 + e⁻ˣ) has derivative σ(x)(1 − σ(x)), never more than 0.25. Chain many of them and the product shrinks: vanishing gradients.
- Central differences are a cheap, reliable check of any derivative you work out by hand.
Common mistakes
- Forgetting the inner rate: the derivative of (3x + 1)² is not just 2(3x + 1).
- Using the wrong values in the chain. Each rate is worked out at the current point (u = 4 here).
- Numerical derivatives with h far too small, like 1e-15: rounding errors become bigger than the answer.
Not true: Numerical derivative: the smaller the step h, the more accurate the answer.
In fact: Only up to a point. With h as small as 1e-15, rounding errors are bigger than the tiny difference, and the answer becomes wrong.
Not true: d/dx (3x + 1)² = 2(3x + 1).
In fact: The chain rule also multiplies by the inner rate, which is 3. The derivative is 2(3x + 1) × 3 = 6(3x + 1).
Part of Unit 1: Math foundations in the free ML Foundations course.