Pretraining data and scaling laws
Pretraining is next-token prediction on an enormous text collection, and scaling laws describe how the resulting loss falls smoothly and predictably as model size, data and compute grow.
Why it matters
Training a large model costs millions, so you need to predict, before spending, how big to make it and how much data to use. Scaling laws let labs fit small experiments and extrapolate. They also explain the shape of the field: why models kept growing, why data quality became the bottleneck, and why small models are now trained on far more data than "optimal".
The idea
Pretraining is next-token prediction on a huge text collection. Scaling laws say the loss falls smoothly and predictably as model size, data and compute grow. The Chinchilla result: for a fixed compute budget, use roughly 20 training tokens per parameter.
Learning a language by reading a whole library. More books and a bigger brain both help, in a predictable way.
Reading the same book a thousand times does not help. Data quality and variety matter as much as size.
How it works
- Collect web pages, books, code and papers, then filter them hard: remove duplicates, spam and low-quality pages, and personal data.
- Tokenise everything and train a GPT on next-token prediction, usually for one pass (or a few) over the data.
- Measure the final loss for many runs of different sizes. Plotted against compute on log–log axes, the points fall on a smooth power-law curve.
- Compute ≈ 6 × parameters × tokens (in floating-point operations). For a fixed compute budget, you choose how to split it between parameters and tokens.
- Chinchilla's result: grow both together, about 20 training tokens per parameter. A 1B-parameter model "wants" about 20B tokens.
- In practice, models meant to be served cheaply are trained far past that point (Llama-style models use trillions of tokens): a smaller, longer-trained model costs less to run.
Key points
- Loss falls as a power law in parameters, data and compute.
- Training compute is about 6·N·D floating-point operations, for a model with N parameters trained on D tokens.
- Compute-optimal: roughly 20 tokens per parameter.
- Data quality, deduplication and mixture matter as much as size.
- Lower loss doesn't map neatly to every ability; some skills seem to appear suddenly with scale.
- Running out of high-quality text is a real constraint; synthetic data is one response.
Common mistakes
- Assuming more data always helps, even when it is duplicated or low quality.
- Reading scaling laws as guarantees for specific skills rather than for the average loss.
- Comparing models trained on different data as if size were the only difference.
Not true: More training text improves a model even when much of it is duplicated or low quality.
In fact: Duplicates and poor text waste compute and can teach bad habits. Filtering the text and removing repeats is a large part of building a pretraining set.
Not true: Fixed compute budget: put it into the biggest model that fits, and train on what data is left.
In fact: A model that is too big for its data ends up under-trained. The compute-optimal balance is roughly 20 training tokens for every parameter.
Part of Unit 4: Transformers and LLMs in the free ML Foundations course.