Autograd and computational graphs

Autograd records every operation into a computational graph as your code runs, then loss.backward() walks the graph in reverse to fill in every gradient automatically.

Why it matters

Writing backward passes by hand is slow and error-prone, and real models have thousands of operations. Autograd removes that work: you write only the forward computation and get exact gradients for free. Knowing how it works explains PyTorch's rules — why gradients add up, why you call zero_grad(), why no_grad() saves memory.

The idea

Autograd records every operation in a graph while you compute. Calling backward() walks the graph in reverse and applies the chain rule for you.

Breadcrumbs. On the way forward you drop a crumb at each step. On the way back you follow them home.

Breadcrumbs cost nothing. The graph costs memory: every intermediate value is stored until backward(). That is why training needs much more memory than inference.

How it works

  1. Mark the tensors that need gradients (requires_grad=True); model parameters already are.
  2. As each operation runs, autograd creates a node that remembers its inputs and how to compute its local derivative.
  3. For L = (a·b + c)·f with a = 2, b = −3, c = 10, f = −2: e = a·b = −6, d = e + c = 4, L = d·f = −8. Each line adds a node to the graph.
  4. loss.backward() starts at L with gradient 1 and visits the nodes in reverse order.
  5. Multiplication gives each input the other input's value times the incoming gradient: d gets f = −2 and f gets d = 4.
  6. Addition passes the gradient through unchanged: e and c both get −2. Then a gets −2·b = 6 and b gets −2·a = −4.
  7. Each result lands in the tensor's .grad, and is added to whatever is already there.

Key points

Common mistakes

Not true: Gradients still flow through a value after .item() or .numpy() has been called on it.
In fact: Those calls leave PyTorch and return plain numbers, so the graph is cut there and nothing before that point receives a gradient.

Not true: PyTorch clears the old gradients by itself before each backward pass.
In fact: Gradients accumulate: each backward pass adds to .grad. Call optimizer.zero_grad() once per step, or the old gradients add up.

Part of Unit 3: Deep learning foundations in the free ML Foundations course.