Pretraining data and scaling laws

Pretraining is next-token prediction on an enormous text collection, and scaling laws describe how the resulting loss falls smoothly and predictably as model size, data and compute grow.

Why it matters

Training a large model costs millions, so you need to predict, before spending, how big to make it and how much data to use. Scaling laws let labs fit small experiments and extrapolate. They also explain the shape of the field: why models kept growing, why data quality became the bottleneck, and why small models are now trained on far more data than "optimal".

The idea

Pretraining is next-token prediction on a huge text collection. Scaling laws say the loss falls smoothly and predictably as model size, data and compute grow. The Chinchilla result: for a fixed compute budget, use roughly 20 training tokens per parameter.

Learning a language by reading a whole library. More books and a bigger brain both help, in a predictable way.

Reading the same book a thousand times does not help. Data quality and variety matter as much as size.

How it works

  1. Collect web pages, books, code and papers, then filter them hard: remove duplicates, spam and low-quality pages, and personal data.
  2. Tokenise everything and train a GPT on next-token prediction, usually for one pass (or a few) over the data.
  3. Measure the final loss for many runs of different sizes. Plotted against compute on log–log axes, the points fall on a smooth power-law curve.
  4. Compute ≈ 6 × parameters × tokens (in floating-point operations). For a fixed compute budget, you choose how to split it between parameters and tokens.
  5. Chinchilla's result: grow both together, about 20 training tokens per parameter. A 1B-parameter model "wants" about 20B tokens.
  6. In practice, models meant to be served cheaply are trained far past that point (Llama-style models use trillions of tokens): a smaller, longer-trained model costs less to run.

Key points

Common mistakes

Not true: More training text improves a model even when much of it is duplicated or low quality.
In fact: Duplicates and poor text waste compute and can teach bad habits. Filtering the text and removing repeats is a large part of building a pretraining set.

Not true: Fixed compute budget: put it into the biggest model that fits, and train on what data is left.
In fact: A model that is too big for its data ends up under-trained. The compute-optimal balance is roughly 20 training tokens for every parameter.

Part of Unit 4: Transformers and LLMs in the free ML Foundations course.