Design an ML training and feature pipeline

The platform that turns raw data into trained, deployed and monitored models, again and again.

Why interviewers ask it

It tests whether you see a model as a product with a lifecycle — data, features, training, deployment and monitoring — repeated forever.

Functional requirements

Non-functional requirements

From version 1 to the final design

  1. Version 1: a notebook trains a model, and someone copies the file to a server.
  2. Version 2: scripts in a scheduled pipeline, and a model registry.
  3. Version 3: data validation, a feature store and evaluation gates.
  4. Version 4: canary releases, drift monitoring, automatic retraining and full lineage.

The design, step by step

  1. An orchestrated pipeline (for example Airflow): ingest → validate the data → build features → train → compare with the current model → register.
  2. A feature store: one definition per feature, an offline store for training (with point-in-time correct joins) and an online store for fast serving.
  3. Deploy with shadow or canary releases; monitor data drift, prediction drift and business metrics.
  4. Retrain on a schedule, or when drift is detected.

Deep dives

Common mistakes

Not true: Writing the feature code twice, once for training and once for serving, is harmless if both follow the same idea.
In fact: Two versions drift apart, and the model then sees different numbers in production from those it trained on. Compute each feature once and use it in both places.

In the interview

An orchestrated pipeline validates data, builds features, trains, compares against the production model and registers the winner with full lineage. A feature store keeps one definition per feature for both training (point-in-time correct) and serving (low latency). New models roll out through shadow or canary releases, are monitored for drift, and are retrained on a schedule or on a trigger.

Part of Unit 9: AI and ML system design in the free System Design course.