Dataset engineering for fine-tuning

Dataset engineering is designing, collecting, cleaning and checking the examples you fine-tune on, because a fine-tuned model learns exactly what its data shows, mistakes included.

Why it matters

When you fine-tune, the data is the product. A few hundred excellent, varied examples often beat thousands of messy ones, and every bad example — rude, wrong, duplicated — is actively taught. Most fine-tuning failures are data failures, so the data deserves most of the effort.

The idea

When you fine-tune, the dataset is the product. Dataset engineering means deciding which examples you need, collecting or generating them, cleaning and de-duplicating them, and checking their quality. A few hundred excellent examples often beat thousands of messy ones. Synthetic data, written by a stronger model, is common, but it must be checked.

Training a new chef with recipe cards. A small box of correct, varied recipes teaches more than a huge pile full of typos, repeats and burnt examples.

A chef can spot a bad recipe and ignore it. A model copies whatever is in the data, mistakes included, so every bad example is actively taught.

How it works

  1. Define the target behaviour precisely, with a few ideal examples written by hand.
  2. Collect candidates: real historical data, human-written examples, and synthetic examples from a stronger model.
  3. Remove exact and near duplicates.
  4. Filter for quality with rules, a model-based grader and human review of a sample. Drop wrong, rude, outdated or off-policy examples.
  5. Balance across topics and difficulty, so rare cases are not lost among the common ones.
  6. Format every example in the model's chat template, consistently.
  7. Hold back a test set the model never trains on, and check that no test example leaks into training.

Key points

Common mistakes

Not true: Raw production logs make a good fine-tuning set just as they are.
In fact: Logs contain wrong, rude and outdated answers, and every one of them would be taught to the model. Filter and check the examples first.

Not true: Thousands of similar examples beat a few hundred varied ones.
In fact: For fine-tuning, quality and variety beat quantity. A few hundred clean, varied examples teach more than thousands of near-copies.

Part of Unit 5: AI engineering in the free ML Foundations course.