Activation functions, and why non-linearity matters
An activation function is the non-linear step applied after each layer's weighted sum; without it, any number of layers collapses into a single linear layer.
Why it matters
The whole point of depth is to build curved, complicated functions. But a matrix times a matrix is just another matrix, so stacked linear layers can never do more than one linear layer. The activation is the small non-linear bend that makes depth useful. Your choice also decides whether gradients flow well through many layers.
The idea
Without a non-linear activation, stacking layers is useless: two linear layers equal one linear layer. ReLU(x) = max(0, x) is the default. Sigmoid and tanh squash values. GELU is a smooth ReLU used in transformers.
Straight Lego bricks can only build straight walls. Activations are the hinged pieces that let you build curves.
Lego hinges are few and fixed. A network learns where to bend, and with enough neurons it can approximate almost any shape.
How it works
- Two linear layers: W2(W1x) = (W2W1)x, which is one matrix. No matter how many you stack, the result is a straight-line model.
- Put a ReLU between them: W2·ReLU(W1x). Each hidden neuron now switches on only above its own threshold, adding one bend where it does.
- Add more neurons, get more bends. With enough of them, a network can follow almost any smooth curve: the universal approximation idea.
- ReLU(z) = max(0, z): cheap, with slope exactly 1 when active, so gradients pass straight through. The default for hidden layers.
- Sigmoid squashes into 0 to 1, and tanh into −1 to 1. Both flatten at the ends, where their slope is nearly 0, so stacking many of them makes gradients vanish. They survive as gates (LSTMs) and as output squashers.
- GELU and SiLU are smooth versions of ReLU that let small negative values through. Transformers use them.
Key points
- No activation means no depth: the whole stack is one linear map.
- ReLU is the default for hidden layers. GELU or SiLU in transformers.
- Sigmoid for yes/no outputs and gates; softmax for picking one of several classes.
- Saturating functions (sigmoid, tanh) give tiny gradients at their ends.
- "Dead ReLUs": a neuron whose input is always negative outputs 0 and gets no gradient, so it never recovers. Big learning rates make this more likely.
- Each ReLU neuron adds a hinge, so the network's output is a piecewise-linear function.
Common mistakes
- Using sigmoid in every hidden layer of a deep network.
- Forgetting the activation in a hand-written layer.
- Putting ReLU on a regression output that should be able to go negative.
Not true: A ReLU on the last layer suits a regression output that can be negative.
In fact: ReLU cannot output a negative number, so a target that can be negative could never be predicted. Leave the last layer of a regression without an activation.
Not true: Hidden layers of a deep network: sigmoid is a sound default.
In fact: Sigmoid saturates: at its two ends the gradient is tiny, and through many layers it vanishes. ReLU is the default for hidden layers.
Part of Unit 3: Deep learning foundations in the free ML Foundations course.