Autograd and computational graphs
Autograd records every operation into a computational graph as your code runs, then loss.backward() walks the graph in reverse to fill in every gradient automatically.
Why it matters
Writing backward passes by hand is slow and error-prone, and real models have thousands of operations. Autograd removes that work: you write only the forward computation and get exact gradients for free. Knowing how it works explains PyTorch's rules — why gradients add up, why you call zero_grad(), why no_grad() saves memory.
The idea
Autograd records every operation in a graph while you compute. Calling backward() walks the graph in reverse and applies the chain rule for you.
Breadcrumbs. On the way forward you drop a crumb at each step. On the way back you follow them home.
Breadcrumbs cost nothing. The graph costs memory: every intermediate value is stored until backward(). That is why training needs much more memory than inference.
How it works
- Mark the tensors that need gradients (requires_grad=True); model parameters already are.
- As each operation runs, autograd creates a node that remembers its inputs and how to compute its local derivative.
- For L = (a·b + c)·f with a = 2, b = −3, c = 10, f = −2: e = a·b = −6, d = e + c = 4, L = d·f = −8. Each line adds a node to the graph.
- loss.backward() starts at L with gradient 1 and visits the nodes in reverse order.
- Multiplication gives each input the other input's value times the incoming gradient: d gets f = −2 and f gets d = 4.
- Addition passes the gradient through unchanged: e and c both get −2. Then a gets −2·b = 6 and b gets −2·a = −4.
- Each result lands in the tensor's .grad, and is added to whatever is already there.
Key points
- The graph is rebuilt on every forward pass, so ordinary Python if-statements and loops just work.
- .grad accumulates, so call optimizer.zero_grad() once per step.
- with torch.no_grad(): stops recording. Use it for evaluation and inference: less memory, faster.
- tensor.detach() cuts a value out of the graph.
- The memory cost of training comes largely from the activations saved for backward.
- micrograd (Karpathy) is a complete autograd engine in about 100 lines of Python. Building one is the best way to understand it.
Common mistakes
- Forgetting zero_grad(), so gradients from earlier batches add up.
- Evaluating without no_grad(), wasting memory on a graph you never use.
- Calling .numpy() or .item() in the middle of a computation, which breaks the graph.
- Changing a tensor in place that autograd still needs, which causes an error during backward.
Not true: Gradients still flow through a value after .item() or .numpy() has been called on it.
In fact: Those calls leave PyTorch and return plain numbers, so the graph is cut there and nothing before that point receives a gradient.
Not true: PyTorch clears the old gradients by itself before each backward pass.
In fact: Gradients accumulate: each backward pass adds to .grad. Call optimizer.zero_grad() once per step, or the old gradients add up.
Part of Unit 3: Deep learning foundations in the free ML Foundations course.