Design an ML training and feature pipeline
The platform that turns raw data into trained, deployed and monitored models, again and again.
Why interviewers ask it
It tests whether you see a model as a product with a lifecycle — data, features, training, deployment and monitoring — repeated forever.
Functional requirements
- Collect and label data; compute features.
- Train, evaluate and register models; deploy them.
- Monitor them and retrain.
Non-functional requirements
- Reproducible: every model can be traced back to its data and code.
- The same features in training and serving.
- Safe rollouts.
From version 1 to the final design
- Version 1: a notebook trains a model, and someone copies the file to a server.
- Version 2: scripts in a scheduled pipeline, and a model registry.
- Version 3: data validation, a feature store and evaluation gates.
- Version 4: canary releases, drift monitoring, automatic retraining and full lineage.
The design, step by step
- An orchestrated pipeline (for example Airflow): ingest → validate the data → build features → train → compare with the current model → register.
- A feature store: one definition per feature, an offline store for training (with point-in-time correct joins) and an online store for fast serving.
- Deploy with shadow or canary releases; monitor data drift, prediction drift and business metrics.
- Retrain on a schedule, or when drift is detected.
Deep dives
- Training-serving skew: compute each feature once and use it in both places.
- Point-in-time joins stop future data leaking into training.
- Data validation catches broken upstream data before it trains a bad model.
- Lineage: every model links to its data version, code commit and settings.
Common mistakes
- Different feature code in training and in serving.
- No data validation: a broken upstream feed, with missing values or changed units, quietly trains a bad model.
- No way to roll back a bad model.
Not true: Writing the feature code twice, once for training and once for serving, is harmless if both follow the same idea.
In fact: Two versions drift apart, and the model then sees different numbers in production from those it trained on. Compute each feature once and use it in both places.
In the interview
An orchestrated pipeline validates data, builds features, trains, compares against the production model and registers the winner with full lineage. A feature store keeps one definition per feature for both training (point-in-time correct) and serving (low latency). New models roll out through shadow or canary releases, are monitored for drift, and are retrained on a schedule or on a trigger.
Part of Unit 9: AI and ML system design in the free System Design course.