RNNs, and why long sequences are hard

A recurrent neural network (RNN) reads a sequence one step at a time. It keeps a hidden state as its memory, updates it at every step, and uses the same weights each time.

Why it matters

Text, audio and time series come in order, with any length. A fixed-size MLP can't handle that, and it can't share what it learns across positions. RNNs were the first neural answer to sequences, and understanding their limits — fading memory and vanishing gradients through time — explains why LSTMs and then transformers took over.

The idea

An RNN reads a sequence one step at a time and keeps a hidden state as memory: hₜ = tanh(W hₜ₋₁ + U xₜ). The same weights are used at every step.

Reading a book with one sticky note. After each page, you rewrite the note.

One small note cannot hold a whole book. Early information gets overwritten, and gradients shrink over many steps. Transformers, which look at every page at once, replaced RNNs for this reason.

How it works

  1. Start with a hidden state h₀ = 0.
  2. At each step t, combine the previous memory and the new input: hₜ = tanh(W·hₜ₋₁ + U·xₜ + b).
  3. The same W, U and b are used at every step, so the model handles any length with a fixed number of weights.
  4. To predict, read an output from each hₜ (the next character, say), or only from the last one (a sentiment score).
  5. Training unrolls the network through time and runs backprop through every step: backpropagation through time.
  6. The gradient reaching step 1 has been multiplied by W (and tanh's slope) once per step. Over 100 steps, a factor of 0.9 becomes 0.9¹⁰⁰ ≈ 0.00003: it vanishes. A factor above 1 explodes instead.
  7. So plain RNNs struggle to connect events far apart. Gradient clipping tames explosions; LSTMs (later in this unit) tame vanishing.

Key points

Common mistakes

Not true: A plain RNN easily remembers a word it read hundreds of steps earlier.
In fact: Its gradient is multiplied by nearly the same factor at every step, so it vanishes or explodes over long distances. LSTMs and attention were invented for that.

Not true: Hidden state: carry it over from one unrelated sequence to the next.
In fact: The hidden state is a summary of what was just read. Reset it (or detach it) between unrelated sequences, or one text leaks into the next.

Part of Unit 3: Deep learning foundations in the free ML Foundations course.