ML system design: pipelines, training vs serving, drift
ML system design is building everything around a model — data collection, labelling, features, training, serving, monitoring and retraining — so it keeps working in production as the world changes.
Why it matters
In production the model is a small part of the system. Most failures come from the data and pipelines: features computed differently in training and serving, labels arriving late, and the world drifting away from the training data. A model that was 95% accurate at launch can quietly decay with no error message.
The idea
A real ML system is much more than a model: data collection, labelling, features, training pipelines, serving, monitoring and retraining. Training-serving skew means the model sees different data in production than in training. Drift means the world changes over time.
A restaurant is not just a recipe. It needs suppliers, a kitchen and waiters, and it must change the menu when customers' tastes change.
A restaurant notices when customers stop coming. A model can fail silently. You only notice if you monitor the right numbers.
How it works
- Define the goal and the metric that matters for the business, and a simple baseline.
- Collect and label data, version it, and split it by time when the future matters.
- Build features with one shared pipeline (or a feature store) used by both training and serving, to avoid training–serving skew.
- Train, evaluate on held-out data, and compare against the current production model.
- Deploy carefully: shadow mode or a small percentage of traffic first, with a rollback plan.
- Monitor inputs (data drift), predictions, outcomes (when labels arrive), latency and errors.
- Retrain on a schedule or when monitoring shows drift, and repeat the evaluation before each release.
Key points
- Training–serving skew: production features differ from training features.
- Data drift: the input distribution changes. Concept drift: the relationship between inputs and outputs changes.
- Monitor what you can see straight away (inputs, predictions) because true labels may arrive late.
- Roll out gradually and keep the ability to roll back.
- Most of the work is data and pipelines, not the model.
Common mistakes
- Computing features one way in training and another way in serving.
- Testing on randomly shuffled data when the real use is predicting the future.
- Having no monitoring after launch, so nobody notices when the data drifts and the model quietly gets worse.
- Retraining automatically without evaluating the new model first.
Not true: A freshly retrained model can go straight into production, because newer data makes it better.
In fact: New data can be broken or unrepresentative, so a retrained model can be worse. Evaluate it against the current one before it takes over.
Not true: Testing a model that will predict the future: a random shuffle of the data is fair.
In fact: A random split lets the model train on later data and be tested on earlier data. Split by time: train on the past, test on what came after.
Part of Unit 5: AI engineering in the free ML Foundations course.