Dataset engineering for fine-tuning
Dataset engineering is designing, collecting, cleaning and checking the examples you fine-tune on, because a fine-tuned model learns exactly what its data shows, mistakes included.
Why it matters
When you fine-tune, the data is the product. A few hundred excellent, varied examples often beat thousands of messy ones, and every bad example — rude, wrong, duplicated — is actively taught. Most fine-tuning failures are data failures, so the data deserves most of the effort.
The idea
When you fine-tune, the dataset is the product. Dataset engineering means deciding which examples you need, collecting or generating them, cleaning and de-duplicating them, and checking their quality. A few hundred excellent examples often beat thousands of messy ones. Synthetic data, written by a stronger model, is common, but it must be checked.
Training a new chef with recipe cards. A small box of correct, varied recipes teaches more than a huge pile full of typos, repeats and burnt examples.
A chef can spot a bad recipe and ignore it. A model copies whatever is in the data, mistakes included, so every bad example is actively taught.
How it works
- Define the target behaviour precisely, with a few ideal examples written by hand.
- Collect candidates: real historical data, human-written examples, and synthetic examples from a stronger model.
- Remove exact and near duplicates.
- Filter for quality with rules, a model-based grader and human review of a sample. Drop wrong, rude, outdated or off-policy examples.
- Balance across topics and difficulty, so rare cases are not lost among the common ones.
- Format every example in the model's chat template, consistently.
- Hold back a test set the model never trains on, and check that no test example leaks into training.
Key points
- For a fine-tuning dataset, quality and variety beat quantity: a few hundred clean, varied examples beat thousands of messy ones.
- Synthetic data is useful but must be filtered and checked.
- De-duplicate, balance and hold out a clean test set.
- Read samples of the final dataset yourself.
- Version datasets like code, so results can be reproduced.
Common mistakes
- Training on raw logs without filtering.
- Test examples leaking into training, so scores lie.
- Synthetic data that copies the generator's mistakes and style.
- Formatting the examples inconsistently, so the model learns the mess as well as the task.
Not true: Raw production logs make a good fine-tuning set just as they are.
In fact: Logs contain wrong, rude and outdated answers, and every one of them would be taught to the model. Filter and check the examples first.
Not true: Thousands of similar examples beat a few hundred varied ones.
In fact: For fine-tuning, quality and variety beat quantity. A few hundred clean, varied examples teach more than thousands of near-copies.
Part of Unit 5: AI engineering in the free ML Foundations course.