Before learning any specific algorithm, you need a way to tell whether a model is good. This lecture covers that toolkit. First, how to split data so the score you report is honest. Second, how to choose a model's settings (hyperparameters) with cross-validation. Third, which numbers to report: accuracy, precision, recall, F1, and the PR and ROC curves. All of it serves one goal: avoiding overfitting, where a model memorizes its training data and fails on new data.
After this you canData vocabulary
A dataset is a collection of samples (also called instances). One sample might be one image, one email, or one patient. Each sample is described by numbers called features (or attributes), for example its color, size or material. Stack a sample's features into a list and you get its feature vector. The number is the dimensionality.
- number of samples in the dataset
- number of features per sample (dimensionality)
- feature of sample
In classification the label is one of a few discrete values. With two classes (spam / not spam) it's binary classification, and we call one class positive and the other negative. With more than two (cat / dog / bird) it's multi-class classification.
"Positive" doesn't mean "good". It means "the class we're looking for". In a cancer test, positive = has cancer. In a spam filter, positive = spam.
Train, validation, test
A model that scores well on the data it learned from proves nothing, because it may simply have memorized it. To get an honest score, you hold some data back. The simplest version is a hold-out split: train on about 70% and test on the remaining 30%. With very little data, 90/10 is fine too, since you need as much as possible for training.
There is a second thing to learn besides the model itself. Compare:
| Parameters | Hyperparameters | |
|---|---|---|
| What | numbers the algorithm learns | settings you pick before training |
| Example | the slope and intercept of a fitted line | in k-nearest neighbors; a learning rate |
| Chosen using | training data | validation data (cross-validation) |
To choose hyperparameters fairly, split the training portion again and set aside a validation set. Try each candidate setting, score it on the validation set, and keep the best. The test set stays locked away until the very end.
The lecture sometimes says "testing" when it means "validation". They are different. If you choose hyperparameters by looking at test scores, the test set has helped train the model, and its score will be too optimistic.
k-fold cross-validation
One validation split is a gamble: you might happen to get an easy or a hard 20%. k-fold cross-validation removes the luck by giving every sample a turn in the validation set.
- Split the training data into equal chunks, called folds. Typical choices are or .
- Round 1: train on folds 2…k and measure the error on fold 1.
- Repeat, holding out each fold once. That's rounds, trained models and error numbers.
- Report the average .
This is how you pick a hyperparameter, which is the main reason the professor introduced it. Run the full k-fold procedure once per candidate value and keep the value with the lowest average error.
Candidate values 0.1, 0.2, 0.3. Run 5-fold CV for each:
| Value | E₁ | E₂ | E₃ | E₄ | E₅ | Average |
|---|---|---|---|---|---|---|
| 0.1 | 0.12 | 0.15 | 0.11 | 0.14 | 0.13 | 0.13 |
| 0.2 | 0.10 | 0.09 | 0.12 | 0.11 | 0.08 | 0.10 ← lowest |
| 0.3 | 0.14 | 0.12 | 0.15 | 0.13 | 0.16 | 0.14 |
Pick 0.2. Then retrain on all the training data with 0.2, and only then evaluate once on the test set. Cost: 3 values × 5 folds = 15 training runs.
With 1,000 training samples and 10-fold CV, how many models get trained, and on how many samples each?
10 models (one per fold). Each fold has 100 samples, so each model trains on the other 9 folds: 900 samples.
Bootstrapping
Bootstrapping is another way to make a train/validation split. From a dataset of samples, draw one sample at random, copy it into a new set , and put it back. Repeat times. This is called sampling with replacement. Every sample has the same chance on every draw.
Because samples go back into the pool, has the same size as but contains duplicates, and some samples never get picked at all. Those leftovers, called out-of-bag samples, become the validation set.
How many get left out? On a single draw, a particular sample is not chosen with probability . The draws are independent, so it misses all of them with probability:
| m | P(a sample is never drawn) |
|---|---|
| 3 | 0.296 |
| 10 | 0.349 |
| 100 | 0.366 |
| 1000 | 0.368 |
So about 36.8% of the data (the lecture rounds this to "about 1/3") is left out for validation, while training still uses samples. You can swap each round of cross-validation for a fresh bootstrap sample and average the errors the same way.
Bootstrapping draws with replacement. Without replacement, draws from samples would just give you the whole dataset back.
MSE, error rate, accuracy
You now have a held-out set. What number do you compute on it? That depends on the task. For regression (predicting a number), measure how far off the predictions are:
- the model's prediction for sample
- the true value (ground truth)
- MAE
- mean absolute error, mentioned in the lecture but not on the slide
For classification, a prediction is either right or wrong, so count:
- indicator function: 1 if the condition inside is true, otherwise 0. Summing it counts the matches.
Regression: true , predicted . The errors are .
MSE . MAE .
The single miss of 2 contributes 4 of the 5.25 total in MSE: squaring punishes big misses hard.
Classification: true , predicted . 2 of 5 are wrong, so error rate and accuracy .
The confusion matrix
Accuracy lumps all mistakes together. In binary classification there are two very different kinds of mistake, and the confusion matrix keeps them apart. Rows are what's actually true; columns are what the model predicted.
| Name | Actual | Predicted | In words |
|---|---|---|---|
| TP true positive | + | + | caught it |
| FN false negative | + | − | missed it |
| FP false positive | − | + | false alarm |
| TN true negative | − | − | correctly ignored |
Naming trick: the second letter is what the model said (Positive/Negative). The first letter says whether it was right (True/False).
A "filter" that labels every email "not spam" scores accuracy while catching zero spam. When one class is rare, accuracy looks great and means little. That's why the next metrics exist.
Precision, recall, sensitivity, specificity
Each of these metrics zooms in on one row or one column of the matrix:
| Metric | Question it answers | Spam example |
|---|---|---|
| Precision | When the model says "yes", how often is it right? | 80 / 110 = 0.727 |
| Recall (sensitivity, true positive rate) | Of all real positives, how many did it catch? | 80 / 100 = 0.800 |
| Specificity | Of all real negatives, how many did it correctly leave alone? | 870 / 900 = 0.967 |
Sensitivity is exactly the same formula as recall. Different fields just use different names. Object detection and search use precision/recall. Medicine uses sensitivity/specificity.
Slide 7 describes sensitivity as "the ratio of correct positive samples to the number classified as positive". That sentence describes precision. The slide's formula, TP/(TP+FN), is the correct one: divide by the number actually positive. Trust the formula.
Ask which mistake hurts more. Missing a positive is costly (cancer screening, fraud): maximize recall. A false alarm is costly (sending a real email to spam, convicting someone): maximize precision.
F1 and F-beta
Usually you want one number that balances precision and recall. The F1 score is their harmonic mean:
Why harmonic instead of the ordinary average? The harmonic mean is dragged toward the smaller value, so you can't hide a terrible recall behind a great precision. With and , the ordinary average is 0.55 but .
When one of them matters more, use :
- equal weight; this is F1
- recall counts more (e.g. )
- precision counts more (e.g. )
. (pulled toward the higher recall). (pulled toward the lower precision).
PR and ROC curves
The lecture shows these curves without explaining where they come from, so here is the missing piece. Most classifiers output a score (say, "0.73 likely spam"), and you choose a threshold: everything above it is called positive. Each threshold gives a different confusion matrix. Sweep the threshold from high to low and plot the results, and you get a curve.
Raising the threshold makes the model pickier: precision tends to go up and recall goes down. Lowering it does the reverse. Across these 10 samples: at threshold 0.9 precision is 1.00 and recall is 0.4; at 0.55 both are 0.8; at 0.1 recall is 1.0 but precision drops to 0.5.
Precision–recall (PR) curve
Plot precision (y) against recall (x) for every threshold. A perfect model hugs the top-right corner: a square with area 1. The break-even point is where the curve crosses the diagonal, i.e. where precision = recall. When comparing models, the curve that sits higher, with larger area or a higher break-even point, is better.
ROC curve
The ROC (receiver operating characteristic) curve plots the true positive rate against the false positive rate:
- Perfect classifier: goes straight up to (0, 1) and then across. AUC = 1.
- Random guessing (the lecture's coin flip): the diagonal. AUC = 0.5.
- Below the diagonal: worse than a coin. Flip its predictions and it would be above.
- Comparing models: the curve closer to the top-left, with larger AUC, is better. On slide 10, A > B > C > D.
AUC is the probability that a randomly picked positive gets a higher score than a randomly picked negative. Check it here: there are 5 × 5 = 25 positive–negative pairs, and 21 are ranked correctly. 21/25 = 0.84.
"AUC" means area under whichever curve you're talking about. The lecture uses it for both PR and ROC curves. They give different numbers, so always say which curve. With very imbalanced data (rare positives), the PR curve is usually more informative.
Overfitting
The professor called this "a very important concept". Overfitting is when a model fits its training data extremely well, often perfectly, but performs badly on new data. It happens when the model is too flexible for the amount of data: instead of learning the underlying pattern, it memorizes the noise.
Generalization means how well a model performs on data it hasn't seen. That's the number that matters. Overfitting is a large gap between training performance and test performance.
| Training error | Validation / test error | Diagnosis | |
|---|---|---|---|
| Model A | low | low | good fit |
| Model B | very low | high | overfitting |
| Model C | high | high | underfitting (too simple) |
How to catch it: cross-validation. Measuring on held-out folds exposes a model that only memorized. Later weeks add more tools: simpler models, more data, and regularization (week 11).
Your model gets 99% training accuracy and 71% cross-validation accuracy. What's going on, and what direction should you move?
That's a big train/validation gap, so it's overfitting. Make the model simpler (or more regularized), or get more training data. Don't judge it by the 99%.
Cheat sheet
| Splits | train → fit parameters; validation → choose hyperparameters; test → final score, once |
| k-fold CV | k rounds, each fold validated once; ; typical or |
| Bootstrap | m draws with replacement; left out |
| MSE / MAE | / |
| Accuracy | = 1 − error rate |
| Precision | : of predicted +, fraction correct |
| Recall = sensitivity = TPR | : of actual +, fraction caught |
| Specificity | ; FPR = 1 − specificity = |
| F1 | , harmonic mean |
| Fβ | ; β > 1 favors recall |
| PR curve | precision vs recall over thresholds; break-even where P = R |
| ROC curve | TPR vs FPR; AUC 1 = perfect, 0.5 = random |
| Overfitting | train ≫ test performance; detect with CV |
Practice
Q1You run 10-fold cross-validation on 1,000 samples. How many models are trained, and how many samples does each train on?
Q2You're choosing for k-nearest neighbors. Which data should you use to compare different values of ?
Q3Bootstrap a dataset of samples. What's the probability a particular sample is never drawn?
Q4TP = 40, FP = 10, FN = 20, TN = 930. What is the precision?
Q51% of transactions are fraud. A model predicts "not fraud" for everything. Its accuracy and recall are:
Q6For a cancer screening test, a missed cancer is far worse than a false alarm (which just leads to a follow-up test). Which metric should be high?
Q7Precision = 0.9, recall = 0.1. What is F1?
Q8You want one score that cares more about recall than precision. Which should you report?
Q9A classifier has ROC AUC = 0.45. What does that tell you?
Q10Training error 1%, cross-validation error 25%. Diagnosis?
Problem 1: Using the 10-sample score line in Section 9 with threshold 0.55, build the confusion matrix and compute accuracy, precision, recall, FPR and F1.
Above 0.55: scores 0.95, 0.9, 0.8, 0.7, 0.6. Four are positive (TP = 4) and one, 0.8, is negative (FP = 1).
Below 0.55: 0.5, 0.4, 0.3, 0.2, 0.1. One is positive, 0.4 (FN = 1), and four are negative (TN = 4).
Accuracy . Precision . Recall . FPR . F1 .
This is the marked point on both curves: the break-even point on the PR curve and (0.2, 0.8) on the ROC curve.
Problem 2: A model has precision 0.5 and recall 0.8. Compute F1 and F2. Which is higher, and why?
.
.
F2 is higher because it weights recall more, and recall (0.8) is this model's strength.