Activation functions, and why non-linearity matters

An activation function is the non-linear step applied after each layer's weighted sum; without it, any number of layers collapses into a single linear layer.

Why it matters

The whole point of depth is to build curved, complicated functions. But a matrix times a matrix is just another matrix, so stacked linear layers can never do more than one linear layer. The activation is the small non-linear bend that makes depth useful. Your choice also decides whether gradients flow well through many layers.

The idea

Without a non-linear activation, stacking layers is useless: two linear layers equal one linear layer. ReLU(x) = max(0, x) is the default. Sigmoid and tanh squash values. GELU is a smooth ReLU used in transformers.

Straight Lego bricks can only build straight walls. Activations are the hinged pieces that let you build curves.

Lego hinges are few and fixed. A network learns where to bend, and with enough neurons it can approximate almost any shape.

How it works

  1. Two linear layers: W2(W1x) = (W2W1)x, which is one matrix. No matter how many you stack, the result is a straight-line model.
  2. Put a ReLU between them: W2·ReLU(W1x). Each hidden neuron now switches on only above its own threshold, adding one bend where it does.
  3. Add more neurons, get more bends. With enough of them, a network can follow almost any smooth curve: the universal approximation idea.
  4. ReLU(z) = max(0, z): cheap, with slope exactly 1 when active, so gradients pass straight through. The default for hidden layers.
  5. Sigmoid squashes into 0 to 1, and tanh into −1 to 1. Both flatten at the ends, where their slope is nearly 0, so stacking many of them makes gradients vanish. They survive as gates (LSTMs) and as output squashers.
  6. GELU and SiLU are smooth versions of ReLU that let small negative values through. Transformers use them.

Key points

Common mistakes

Not true: A ReLU on the last layer suits a regression output that can be negative.
In fact: ReLU cannot output a negative number, so a target that can be negative could never be predicted. Leave the last layer of a regression without an activation.

Not true: Hidden layers of a deep network: sigmoid is a sound default.
In fact: Sigmoid saturates: at its two ends the gradient is tiny, and through many layers it vanishes. ReLU is the default for hidden layers.

Part of Unit 3: Deep learning foundations in the free ML Foundations course.