ML system design: pipelines, training vs serving, drift

ML system design is building everything around a model — data collection, labelling, features, training, serving, monitoring and retraining — so it keeps working in production as the world changes.

Why it matters

In production the model is a small part of the system. Most failures come from the data and pipelines: features computed differently in training and serving, labels arriving late, and the world drifting away from the training data. A model that was 95% accurate at launch can quietly decay with no error message.

The idea

A real ML system is much more than a model: data collection, labelling, features, training pipelines, serving, monitoring and retraining. Training-serving skew means the model sees different data in production than in training. Drift means the world changes over time.

A restaurant is not just a recipe. It needs suppliers, a kitchen and waiters, and it must change the menu when customers' tastes change.

A restaurant notices when customers stop coming. A model can fail silently. You only notice if you monitor the right numbers.

How it works

  1. Define the goal and the metric that matters for the business, and a simple baseline.
  2. Collect and label data, version it, and split it by time when the future matters.
  3. Build features with one shared pipeline (or a feature store) used by both training and serving, to avoid training–serving skew.
  4. Train, evaluate on held-out data, and compare against the current production model.
  5. Deploy carefully: shadow mode or a small percentage of traffic first, with a rollback plan.
  6. Monitor inputs (data drift), predictions, outcomes (when labels arrive), latency and errors.
  7. Retrain on a schedule or when monitoring shows drift, and repeat the evaluation before each release.

Key points

Common mistakes

Not true: A freshly retrained model can go straight into production, because newer data makes it better.
In fact: New data can be broken or unrepresentative, so a retrained model can be worse. Evaluate it against the current one before it takes over.

Not true: Testing a model that will predict the future: a random shuffle of the data is fair.
In fact: A random split lets the model train on later data and be tested on earlier data. Split by time: train on the past, test on what came after.

Part of Unit 5: AI engineering in the free ML Foundations course.