The neuron and the multi-layer perceptron
A neuron computes a weighted sum of its inputs plus a bias, and passes it through an activation. A multi-layer perceptron (MLP) stacks layers of neurons, each layer feeding the next.
Why it matters
Linear and logistic regression can only draw straight boundaries and fit straight trends. Real patterns — handwriting, speech, prices driven by many interacting factors — are curved and tangled. Stacking many simple neurons in layers lets a model build complicated functions out of simple pieces, and every modern network, transformers included, is built from this unit.
The idea
A neuron computes a weighted sum of its inputs, adds a bias, and applies an activation: a = f(w·x + b). An MLP stacks layers of neurons, and each layer's output is the next layer's input.
Rows of judges. Each judge (neuron) weighs the evidence in their own way and gives a score. The next row of judges combines those scores.
Real judges can explain their reasoning. Single neurons usually cannot be read so simply.
How it works
- One neuron takes inputs x, multiplies each by its weight, adds them up and adds a bias: z = w·x + b. For x = [0.5, −1, 2], w = [0.8, −0.4, 0.6] and b = −0.5, z = 0.4 + 0.4 + 1.2 − 0.5 = 1.5.
- It applies an activation: ReLU(1.5) = 1.5. That number is the neuron's output.
- A layer is many neurons reading the same inputs, each with its own weights. In code, a whole layer is one matrix multiply: h = relu(x @ W + b).
- Shapes: with a batch of inputs X of shape (batch, 3) and W of shape (3, 4), the layer's output has shape (batch, 4): 4 neurons for each example.
- An MLP feeds each layer's output into the next: input → hidden layers → output layer.
- The last layer has no ReLU: it gives one raw score per class (or one number for regression).
- Count the parameters: a layer from n inputs to m neurons has n × m weights and m biases. A 3 → 4 → 2 network has 12 + 4 + 8 + 2 = 26.
Key points
- Neuron: a = f(w·x + b). Logistic regression is one neuron with a sigmoid.
- Layer: h = f(x W + b), done for the whole batch with one matrix multiply.
- "Fully connected" (or "dense", or "linear") layer: every input connects to every neuron.
- Parameters per layer = n_in × n_out + n_out.
- The forward pass runs the layers in order; training adjusts every weight with backprop.
- Depth (more layers) lets a network build features out of features.
Common mistakes
- Forgetting the activation between layers, so the whole network collapses into one linear layer.
- Getting the weight shape backwards: for x @ W, W must be (n_in, n_out).
- Starting every weight at the same value: every neuron in a layer then stays identical forever.
- Putting a ReLU on the final output, which stops the scores going negative.
Not true: A network learns normally when its weights start at one shared value.
In fact: Every neuron in a layer would then compute the same thing and receive the same update, so they stay identical for ever. Random starting weights break the tie.
Not true: Linear layers stacked with no activation between them: a more powerful network.
In fact: Without an activation between them, several linear layers collapse into one linear layer. The non-linearity is what makes depth useful.
Part of Unit 3: Deep learning foundations in the free ML Foundations course.