GPT-style models and next-token prediction
A GPT is a stack of transformer blocks with a causal mask, trained to predict the next token; it generates text by predicting a token, appending it and repeating.
Why it matters
Next-token prediction needs no labels: any text is its own training data, so models can learn from trillions of tokens. And one simple loop — predict, append, repeat — turns that single skill into writing, answering, summarising and coding. Every chat assistant is built on this.
The idea
A GPT reads tokens left to right and predicts the next token. A causal mask stops each position from seeing the future. Generation means: predict, append, repeat.
The next-word suggestion on your phone keyboard, but trained on a huge amount of text and far deeper.
A phone keyboard looks at a few words. A GPT looks at thousands of tokens of context at once.
How it works
- Tokenise the text and look up embeddings plus positions.
- Run the transformer blocks with a causal mask: position t can attend only to positions up to t.
- At every position, a final linear layer gives logits over the whole vocabulary: the model's guess for the next token.
- Training: the targets are the inputs shifted left by one. "The cat sat on the" trains the predictions cat, sat, on, the — every position at once.
- The loss is cross-entropy between each position's predicted distribution and the true next token, averaged over all positions.
- Generation: feed in the prompt, take the distribution at the last position, choose a token (greedy or sampled), append it, and repeat until a stop token or a length limit.
Key points
- Self-supervised: the text supplies its own labels.
- The causal mask makes training efficient (all positions in parallel) and honest (no peeking at the future).
- Generation is sequential: one forward step per new token.
- The context window is the maximum number of tokens the model can look at.
- The output at each step is a probability distribution; how you choose from it is decoding (next topics).
- Chat assistants are GPTs plus instruction and preference tuning (later in this unit).
Common mistakes
- Training without the causal mask: the loss looks great because the model copies the answer, but generation fails.
- Off-by-one errors when shifting inputs and targets.
- Expecting the model to know facts that weren't in its training text, or after its cut-off date.
Not true: A language model knows about events that happened after its training text was collected.
In fact: It learned only from its training text. Anything newer has to be given to it, in the prompt or through retrieval.
Not true: Training without the causal mask: a better model, because the loss is much lower.
In fact: The loss is low only because the model can see the answer and copy it. When it generates, there is no future to copy from, so it fails.
Part of Unit 4: Transformers and LLMs in the free ML Foundations course.