Encoder models (BERT) vs decoder models (GPT)

Encoder models, like BERT, let every token look in both directions and learn by filling in masked words, which suits understanding text. Decoder models, like GPT, look only backwards and learn by predicting the next word, which suits writing it.

Why it matters

Different jobs need different views of the text. Classifying a review or building a search embedding benefits from seeing the whole sentence at once. Writing text one word at a time needs a model that never peeks ahead. Knowing which family a model belongs to tells you what it is good for and how to use it.

The idea

Encoder models like BERT see the whole sentence in both directions and are trained to fill in masked words. They are good for understanding: classification and search embeddings. Decoder models like GPT see only the past and are trained to predict the next word. They are good for generating text.

BERT does fill-in-the-blank exercises. GPT writes a story one word at a time.

The line is blurry today. Decoder models are also used for classification and embeddings.

How it works

  1. Both families use the same transformer blocks. The difference is the attention mask and the training goal.
  2. Encoder: no causal mask, so each token sees the whole input, left and right.
  3. BERT's training: hide about 15% of tokens with [MASK] and predict them from the surrounding words (masked language modelling).
  4. Decoder: a causal mask, so each token sees only earlier tokens. Training: predict the next token.
  5. Encoder–decoder models (the original transformer, T5) use an encoder to read the input and a decoder that generates while attending to it through cross-attention. Common for translation.
  6. Today most large models are decoder-only, and they are also used for classification and embeddings, but encoders remain cheap and strong for search and classification.

Key points

Common mistakes

Not true: A decoder model such as GPT cannot be used for classification.
In fact: It can: ask for the label in the prompt, or put a small classification head on top. An encoder is simply the more efficient choice for the job.

Not true: Encoder (BERT) = the natural choice for writing long text.
In fact: BERT is trained to fill in hidden words while seeing both sides, not to continue a text. Writing one word after another is the job of a decoder such as GPT.

Part of Unit 4: Transformers and LLMs in the free ML Foundations course.