The neuron and the multi-layer perceptron

A neuron computes a weighted sum of its inputs plus a bias, and passes it through an activation. A multi-layer perceptron (MLP) stacks layers of neurons, each layer feeding the next.

Why it matters

Linear and logistic regression can only draw straight boundaries and fit straight trends. Real patterns — handwriting, speech, prices driven by many interacting factors — are curved and tangled. Stacking many simple neurons in layers lets a model build complicated functions out of simple pieces, and every modern network, transformers included, is built from this unit.

The idea

A neuron computes a weighted sum of its inputs, adds a bias, and applies an activation: a = f(w·x + b). An MLP stacks layers of neurons, and each layer's output is the next layer's input.

Rows of judges. Each judge (neuron) weighs the evidence in their own way and gives a score. The next row of judges combines those scores.

Real judges can explain their reasoning. Single neurons usually cannot be read so simply.

How it works

  1. One neuron takes inputs x, multiplies each by its weight, adds them up and adds a bias: z = w·x + b. For x = [0.5, −1, 2], w = [0.8, −0.4, 0.6] and b = −0.5, z = 0.4 + 0.4 + 1.2 − 0.5 = 1.5.
  2. It applies an activation: ReLU(1.5) = 1.5. That number is the neuron's output.
  3. A layer is many neurons reading the same inputs, each with its own weights. In code, a whole layer is one matrix multiply: h = relu(x @ W + b).
  4. Shapes: with a batch of inputs X of shape (batch, 3) and W of shape (3, 4), the layer's output has shape (batch, 4): 4 neurons for each example.
  5. An MLP feeds each layer's output into the next: input → hidden layers → output layer.
  6. The last layer has no ReLU: it gives one raw score per class (or one number for regression).
  7. Count the parameters: a layer from n inputs to m neurons has n × m weights and m biases. A 3 → 4 → 2 network has 12 + 4 + 8 + 2 = 26.

Key points

Common mistakes

Not true: A network learns normally when its weights start at one shared value.
In fact: Every neuron in a layer would then compute the same thing and receive the same update, so they stay identical for ever. Random starting weights break the tie.

Not true: Linear layers stacked with no activation between them: a more powerful network.
In fact: Without an activation between them, several linear layers collapse into one linear layer. The non-linearity is what makes depth useful.

Part of Unit 3: Deep learning foundations in the free ML Foundations course.