Unit 3: Deep learning foundations
Build neural networks from a single neuron up. Write backprop by hand first. Then you can trust PyTorch, because you know what it does.
The big picture
A neural network is many small logistic regressions stacked in layers and trained together with backprop. Unit 2 made the bricks. Unit 3 builds the wall.
Lessons in this unit
- The neuron and the multi-layer perceptron: A neuron computes a weighted sum of its inputs plus a bias, and passes it through an activation. A multi-layer perceptron (MLP) stacks layers of neurons, each layer feeding the next.
- Activation functions, and why non-linearity matters: An activation function is the non-linear step applied after each layer's weighted sum; without it, any number of layers collapses into a single linear layer.
- Backpropagation by hand: Backpropagation computes the gradient of the loss with respect to every weight by applying the chain rule from the output backwards, reusing each intermediate result.
- Autograd and computational graphs: Autograd records every operation into a computational graph as your code runs, then loss.backward() walks the graph in reverse to fill in every gradient automatically.
- Softmax and cross-entropy loss: Softmax turns a vector of raw scores (logits) into probabilities that sum to 1. Cross-entropy loss is −log of the probability given to the correct class. Together, their gradient is simply p − y.
- Optimisers: SGD, momentum, Adam; learning-rate schedules: An optimiser turns gradients into weight updates. SGD steps straight down the gradient, momentum adds velocity, and Adam also adapts the step size for every weight. A learning-rate schedule changes the step size during training.
- Weight initialisation; vanishing and exploding gradients: Weight initialisation chooses the random starting weights so that signals and gradients keep roughly the same size from layer to layer, instead of vanishing towards zero or exploding towards infinity.
- Batch norm and layer norm: A normalisation layer rescales activations to mean 0 and variance 1, then lets the network scale and shift them again. Batch norm does this for each feature across the batch; layer norm does it for each example across its features.
- Regularisation: dropout, weight decay, augmentation, early stopping: Regularisation in deep learning is any trick that stops a network memorising its training data: dropout, weight decay, data augmentation and early stopping are the main ones.
- PyTorch: nn.Module, DataLoader and the training loop: In PyTorch, an nn.Module holds the layers and weights, a DataLoader serves shuffled mini-batches, and the training loop repeats the same few steps: forward, loss, zero_grad, backward, step.
- Convolutional networks: convolution, pooling, receptive field: A convolutional layer slides small learned filters across an image, computing a dot product at every position, so the same pattern detector is reused everywhere. Stacking layers builds up from edges to whole objects.
- Residual connections (ResNet): A residual connection adds a block's input to its output. The block then only has to learn a change, and gradients get a direct path back through the network.
- RNNs, and why long sequences are hard: A recurrent neural network (RNN) reads a sequence one step at a time. It keeps a hidden state as its memory, updates it at every step, and uses the same weights each time.
- Embeddings: representing things as vectors: An embedding gives each item, such as a word, a token, a user or a product, a learned vector of numbers, stored as one row of a table. Similar items end up with vectors that are close together.
- Debugging training: overfit one batch, read loss curves: Debugging training means checking, in a fixed order, that the model can learn at all — sensible starting loss, overfitting one small batch — before reading the loss curves of real runs.
- LSTMs and GRUs: gates for long memory: An LSTM is a recurrent cell with a separate memory line, the cell state, and three learned gates that decide what to erase, what to write and what to reveal. A GRU does the same job with two gates.