Unit 2: Classical ML, built by you
Build every classic algorithm yourself in NumPy. After this unit, a library is only a shortcut for something you already understand.
The big picture
Classical ML is a toolbox of different tools for one job: learning a pattern from examples. Build each tool once by hand, and you will know which one to pick, and why.
Lessons in this unit
- The ML setup: features, labels, loss, train/validation/test: Supervised learning turns examples, each with features and a label, into a model. You fit it on a training set, make your choices on a validation set, and measure the final result once on a test set.
- Linear regression: normal equation and gradient descent: Linear regression predicts a number as a weighted sum of the features. It chooses the weights that make the mean squared error as small as possible.
- Logistic regression and the sigmoid: Logistic regression answers yes/no questions. It computes a score, squashes it into a probability with the sigmoid, and is trained with cross-entropy loss.
- Regularisation: L1 and L2: Regularisation adds a penalty for large weights to the loss. L2 shrinks every weight, while L1 pushes the weak ones to exactly zero.
- Bias–variance trade-off and overfitting: Bias is error from a model too simple to capture the pattern. Variance is error from a model so flexible that it chases the noise in its training data. Good models balance the two.
- Metrics: precision, recall, F1, ROC-AUC, confusion matrix: Classification metrics count the four outcomes (true and false positives and negatives) and summarise them: precision, recall, F1, accuracy, and ROC-AUC across all thresholds.
- Cross-validation and data leakage: Cross-validation scores a model by rotating which slice of the data is held out, so every row is tested once. Data leakage is any way that information from the held-out data sneaks into training.
- Feature scaling and encoding: Feature scaling puts numeric features on similar ranges, and encoding turns categories into numbers, so that models treat every feature fairly and don't invent orders that aren't there.
- k-nearest neighbours: k-nearest neighbours predicts by finding the k training points closest to a new point and letting them vote (or averaging their values, for regression).
- Decision trees: entropy and Gini: A decision tree predicts by asking a sequence of yes/no questions about single features, each chosen to make the resulting groups as pure as possible.
- Ensembles: bagging, random forests, gradient boosting: An ensemble combines many models. Bagging and random forests average trees trained on random resamples, which cuts variance. Boosting adds small trees one after another, each fixing the errors left so far, which cuts bias.
- Support vector machines: margin and hinge loss: A support vector machine separates two classes with the line (or plane) that leaves the widest possible margin to the nearest points, trained with the hinge loss.
- Naive Bayes: Naive Bayes classifies with Bayes' rule, assuming the features are independent once you know the class, so each feature's evidence can simply be multiplied in.
- k-means clustering: k-means groups unlabelled points into k clusters by repeating two steps: assign every point to its nearest centre, then move each centre to the average of its points.
- PCA in practice: In practice, PCA is used to compress features, remove noise, speed up other models and plot high-dimensional data, keeping just enough components to hold most of the variance.
- Gaussian mixture models and EM: A Gaussian mixture model describes data as a blend of several bell curves, and the EM algorithm fits it by alternating soft assignment of points (E-step) with refitting each curve (M-step).
- Generalisation: VC dimension and double descent: Generalisation is doing well on unseen data. The VC dimension measures how much a family of models is able to fit. Double descent is the surprise that test error can fall again once models grow big enough to fit the training data exactly.