LSTMs and GRUs: gates for long memory

An LSTM is a recurrent cell with a separate memory line, the cell state, and three learned gates that decide what to erase, what to write and what to reveal. A GRU does the same job with two gates.

Why it matters

Plain RNNs forget quickly, because the gradient is multiplied by the same factor at every step back in time. LSTMs add a memory path that is updated by adding rather than by repeated multiplication, so information and gradients can survive for hundreds of steps. They powered speech recognition and translation before transformers, and the gating idea lives on in many models.

The idea

An LSTM adds a separate memory line, the cell state, plus three gates that decide what to forget, what to write and what to output. Because the cell state is updated by adding rather than by repeated multiplying, gradients survive over long sequences. A GRU is a simpler version with two gates.

A notebook with a pencil and an eraser. Instead of rewriting the whole sticky note on every page, you erase a few lines (forget gate), write a few new ones (input gate), and read out what you need (output gate).

You decide what to erase on purpose. The gates are learned numbers between 0 and 1, so the network 'half-erases' things. And it still struggles with very long texts that transformers handle easily.

How it works

  1. Inputs at each step: the previous hidden state h, the previous cell state c, and the new input x.
  2. Forget gate f = σ(W_f·[h, x] + b_f): a number from 0 to 1 per memory slot, saying how much of the old memory to keep.
  3. Input gate i = σ(…) and candidate g = tanh(…): what new content to write, and how much of it.
  4. Update the memory by adding: c = f·c_prev + i·g. For c_prev = 0.8, f = 0.9, i = 0.5 and g = 0.6: c = 0.72 + 0.30 = 1.02.
  5. Output gate o = σ(…) chooses what to reveal: h = o·tanh(c) = 0.7 × tanh(1.02) ≈ 0.54.
  6. Backwards, the path from c_prev to c is multiplied only by f, so when f is near 1 the gradient flows back almost unchanged.
  7. A GRU merges the forget and input gates into one update gate and has no separate cell state: fewer parameters, often similar results.

Key points

Common mistakes

Not true: An LSTM carries information across thousands of steps without losing it.
In fact: Its gates leak a little at every step, so very old information still fades. An LSTM remembers far longer than a plain RNN, not for ever.

Not true: The four parts of an LSTM (f, i, g and o) each use a sigmoid.
In fact: The three gates f, i and o use a sigmoid, which gives values from 0 to 1. The candidate g uses tanh, which gives values from −1 to 1.

Part of Unit 3: Deep learning foundations in the free ML Foundations course.