Week 2 showed the failure mode: a flexible model can drive its training error to zero and still predict new data badly (overfitting). This lecture is about the most widely used cure. Instead of minimizing the loss alone, we minimize loss + λ × (size of the weights). The extra term, the regularizer, depends only on the model, never on the data, and the number sets how much we care about it. Measure size with the squared L2 norm and you get ridge regression (weights shrink smoothly, and the "weight decay" update in gradient descent). Measure it with the L1 norm and you get the lasso (many weights become exactly zero, so it also selects features). The same idea appears in the SVM's from week 5, in logistic regression, and in neural networks, where dropout, data augmentation and early stopping join the toolbox.
After this you canOverfitting and three fixes
Slides 6 and 7 show the picture from week 2 three times over: a model that is too simple (underfitting), one that is about right, and one that is too flexible (overfitting). In the language of the bias/variance trade-off:
| Underfit | Good fit | Overfit | |
|---|---|---|---|
| Example (slide 6) | degree 4 through every point | ||
| Training error | high | low | ≈ 0 |
| Error on new data | high | low | high |
| Bias / variance | high bias | balanced | low bias, high variance |
Bias is error from wrong assumptions (a line can't follow a curve). Variance is sensitivity to the particular training sample (a wiggly curve changes completely if you move one point). Overfitting is most likely when there are many features or parameters and few examples. The professor's example is 100 features with only a handful of houses (slide 8).
The lecture lists three ways to fight overfitting (11:00–12:44 and slide 9):
- Get more training data. More points pin the curve down. In deep learning, when real data is scarce, you manufacture more by data augmentation (Section 8).
- Use fewer features. Pick the useful ones by hand, use a model-selection algorithm, or compress them with PCA (week 10's dimension reduction).
- Regularize. Keep all the features, but shrink the size of the parameters. This works well when there are many features that each contribute a little to predicting .
"PCA is trying to reduce the dimensionality. Regularization is trying to squeeze, or narrow down, your feasible space for your model." Regularization doesn't remove any feature; it limits how large the weights may become, so the model can't build the extreme combinations that produce wild curves.
Slide 3 gives a second reason, specific to deep networks: regularization helps "avoid NaN for extreme weights". If weights are free to grow without bound, activations and gradients can overflow to infinity and then to NaN (not-a-number). A small L2 penalty keeps them in a sane range (Section 8).
A model scores 99% on training data and 70% on validation data. Which regularization change would you try first?
That's a big train/validation gap, the signature of overfitting (high variance). Increase the regularization strength (or get more data, or use fewer features). Decreasing would make it worse.
The regularized cost function
Intuition first (slide 10). Suppose the degree-4 model below overfits, and we suspect the culprits are and :
Add a big charge for them to the thing we minimize:
Any sizeable or now costs far more than it could save in squared error, so the minimizer makes them nearly zero. The fitted curve becomes essentially quadratic, which is the good fit. The professor's summary (14:43): a larger coefficient on a parameter's square forces that parameter to be smaller.
In practice we don't know which parameters are the culprits, so we penalize all of them (except the intercept). Small parameters give a "simpler", smoother hypothesis that is less prone to overfitting (slide 12). The general form (slide 2) is
and for linear regression with the L2 regularizer (slides 13–14):
- number of training examples
- number of features; parameters are
- the model's prediction; for linear regression (with )
- regularization parameter: how much small weights matter compared with fitting the data
- the regularizer, ; the sum starts at
The sum starts at (the professor's note on slide 12: "bias, don't regularize bias"). only shifts the curve up or down; shrinking it toward 0 would drag every prediction toward 0 for no gain in smoothness. Géron says the same: "the bias term is not regularized".
The two extremes of λ (17:38–18:58):
- : the penalty disappears and we're back to plain least squares, overfitting included (left panel above).
- huge (slide 14 asks about ): any nonzero costs enormously, so all of them go to ≈ 0. The model becomes , a flat line at the mean of the 's. "Not a good model": severe underfitting.
- In between, is a hyperparameter that trades fit against simplicity. You tune it on validation data (Section 7).
10 training points from (noise standard deviation 0.25), features . Validation error is measured on 51 fresh points. RMSE = root mean squared error.
| λ | train RMSE | validation RMSE | largest |θⱼ| |
|---|---|---|---|
| 0 | 0.000 | 2.556 | 848,000 |
| 10⁻⁶ | 0.138 | 0.249 | 45.0 |
| 10⁻⁵ | 0.143 | 0.241 | 21.2 |
| 10⁻⁴ | 0.146 | 0.242 | 15.6 |
| 0.001 | 0.186 | 0.289 | 8.8 |
| 0.01 | 0.258 | 0.369 | 3.0 |
| 0.1 | 0.327 | 0.410 | 1.2 |
| 1 | 0.477 | 0.526 | 0.541 |
| 10 | 0.622 | 0.648 | 0.122 |
Read it top to bottom. Training error only goes up as grows (the model is more constrained). Validation error falls from 2.56 to about 0.24 and then rises again. The weights tell the story: the unregularized fit needs coefficients near 850,000 that cancel each other to thread through every point, and even cuts them to 45.
Slide 2 writes and with as the number of weights, then uses as the number of examples in on the same slide. In the rest of the deck (and here), = examples and = features, so those sums run to . Also note that "L2 regularization" uses the squared L2 norm, , not .
With and data whose -values average 4, what does the fitted regression line look like?
All slopes are crushed to ≈ 0, and (unpenalized) takes the value that minimizes squared error for a constant model: the mean. So it's the horizontal line . The worked example in Section 4 shows this happening step by step.
Gradient descent: weight decay
Differentiate . The squared-error part gives the familiar gradient from week 3. The penalty adds , but only for :
One gradient-descent step with learning rate , regrouped (slide 17, bottom line):
and is updated exactly as before, with no shrink factor.
This is the "key difference" the professor stresses (19:40–21:06): every step first shrinks each weight by the factor , then takes the ordinary least-squares step. The factor is a little below 1 (the slide's handwritten note: "≈ 0.99"). Because weights decay geometrically toward zero unless the data keeps pushing them up, L2 regularization is also called weight decay (Bishop §3.1.4).
Inside the bracket, the annotation writes
It must be : the derivative of is . With a minus sign, the step would grow the weights. The printed line at the bottom of slide 17, with the factor , is correct, and slide 22 prints .
Data: , , so . Model . Take , . The shrink factor is .
Start at the unregularized least-squares answer . There, the squared-error gradient is exactly zero, so the only force is the shrink:
- Step 1. (no loss gradient, no shrink). .
- Step 2. Predictions ; residuals . Loss gradients: and . So and .
- Keep going. The slope shrinks, the intercept rises to compensate, and the iterates settle at , the point where the shrink and the data's pull balance. Section 4 gets this same answer in one line.
| step | θ₀ | θ₁ |
|---|---|---|
| 0 | 0.5000 | 1.4000 |
| 1 | 0.5000 | 1.3650 |
| 2 | 0.5088 | 1.3571 |
| 3 | 0.5186 | 1.3532 |
| ∞ | 1.0833 | 1.1667 |
With , and , what is the shrink factor? What happens if ?
. If the factor is negative, so each step flips the weight's sign. That's a sign the learning rate (or ) is far too big; with sensible values the factor is just below 1, as the slide's note says.
The normal equation with λ
Stack the examples as rows of (with a leading column of 1s) and the targets in , as in week 3. In matrix form,
where is the identity with a 0 in the top-left corner (so is not penalized). Setting the gradient to zero:
With this is week 3's normal equation. The term is the entire effect of L2 regularization.
Same data as Section 3: , . With one feature, the matrix formula reduces to two short ones. Let , , , . Then
| λ | θ₀ | θ₁ (slope) | prediction at x = 5 | sum of squared errors |
|---|---|---|---|---|
| 0 | 0.500 | 1.400 | 7.500 | 0.200 |
| 1 | 1.083 | 1.167 | 6.917 | 0.472 |
| 5 | 2.250 | 0.700 | 5.750 | 2.650 |
| 20 | 3.300 | 0.280 | 4.700 | 6.472 |
| → ∞ | 4.000 | 0 | 4.000 | 10.000 |
The slope shrinks smoothly toward 0 but never reaches it for any finite . The training error rises as the price. In the limit we get slide 14's flat line at . The row matches where gradient descent converged in Section 3.
The bonus: invertibility
Week 3's formula needs to be invertible. Slide 19 points out when it isn't: if there are no more examples than features (), the matrix has rank at most , so it is singular. The same happens with duplicated or perfectly correlated features. The professor calls this an ill-posed problem (21:30): infinitely many fit equally well, and the formula breaks.
For any , is always invertible. Proof sketch: for any nonzero ,
The second term is positive unless , and then , so the first is positive. A positive-definite matrix is invertible.
Two identical features, no intercept: rows , . Then
The determinant is 0: any fits perfectly, so there is no unique answer. With :
Ridge picks the solution that splits the weight evenly between the twins. As it approaches , the smallest-norm exact fit.
Different sources scale the penalty differently, so "" means different things. The slides use , whose solution is . Bishop (Eq. 3.27–3.28) uses : same formula. scikit-learn's Ridge(alpha) minimizes : again the same, so alpha = the slides' . Géron writes the cost as (Eq. 4-8) but gives the closed form (Eq. 4-9). Those two only agree if the in 4-9 is from 4-8; the closed form is the one that matches scikit-learn. When you compare numbers across sources, check the scaling first.
The lecture calls "the covariance matrix". It's only proportional to the feature covariance matrix when the features are centered (mean 0), and then you divide by . The point being made still holds: when this matrix is low-rank, inverting it is ill-posed, and adding fixes that.
You have 50 examples and 200 features. Can you use the plain normal equation? The ridge one?
Plain: no. , so (201 × 201) has rank at most 50 and is singular. Ridge: yes, for any , is invertible.
L1 vs L2: why lasso gives zeros
Slide 2 offers two ways to measure the size of the weights:
To see how they differ, rewrite "minimize loss + λ·penalty" in an equivalent constrained form: minimize the loss alone, but only over weights inside a region, or . Each corresponds to some budget (Bishop §3.1.4, via Lagrange multipliers): larger λ ↔ smaller region (12:44). The region is the professor's "feasible space". In 2-D, the L2 region is a disk and the L1 region is a diamond (a square turned 45°, which the lecture calls "the square").
Why corners win: the diamond sticks out furthest exactly at its corners, which lie on the axes. An ellipse coming in from a random direction is much more likely to hit a corner than a flat edge. In dimensions the L1 ball has corners and edges on every axis and coordinate plane, so many coordinates end up exactly 0. The disk has no corners: the touching point is almost never on an axis, so ridge weights are small but nonzero "with high probability" (09:02).
"From this geometry perspective, you should remember, you should understand: if you add L1 regularization to your problem, the solution you get will be sparse." And in the closing summary: "L1 regularization gives you a sparse representation or sparse model. L2 regularization gives a non-sparse model." Be ready to draw the diamond and the disk and explain the corner argument.
Watching weights as λ grows
Back to the four points (, ), now with the L1 cost . After the intercept absorbs the means, the slope minimizes
- If : set the derivative to 0: This is only valid while it's positive, i.e. .
- If : at the slope of the squared-error part is , and the kink of can absorb any slope up to in size. So exactly is the minimum.
- Compare to ridge: , which is never 0.
| λ | 0 | 4 | 10 | 14 | 20 |
|---|---|---|---|---|---|
| Lasso θ₁ | 1.400 | 1.000 | 0.400 | 0 | 0 |
| Ridge θ₁ | 1.400 | 0.778 | 0.467 | 0.368 | 0.280 |
The L1 penalty's slope is a constant all the way down to 0, so it keeps pushing a small weight just as hard as a big one, and the weight snaps to 0. The L2 penalty's slope fades as shrinks, so the push dies out before reaching 0.
Training a lasso. has no derivative at 0, but gradient descent still works if you use the subgradient (with sign(0) = 0) in place of the derivative (Géron Eq. 4-11). The step size has to shrink over time, or the weights bounce around zero instead of settling there.
Elastic net: both at once
The lecture (02:12–03:03) mentions using L1 and L2 together, called elastic net. Géron's version mixes them with a ratio :
is lasso, is ridge.
| L2 (ridge) | L1 (lasso) | Elastic net | |
|---|---|---|---|
| Penalty | a mix of both | ||
| Region | disk / ball | diamond (corners on axes) | rounded diamond |
| Weights | all shrunk, none exactly 0 | many exactly 0 (sparse) | some exactly 0 |
| Closed form? | yes (Section 4) | no; subgradient or coordinate descent | no |
| Use when | good default; many small effects | you suspect few features matter | like lasso, but with correlated features or , where lasso can behave erratically |
A model with 1,000 features is trained with lasso and ends with 37 nonzero weights. What has the lasso done besides regularizing?
Feature selection: it decided 963 features aren't worth their penalty and removed them from the model entirely. Ridge would have kept all 1,000 with small weights.
Regularized logistic regression
Logistic regression (week 3's classifier, ) overfits the same way: with many polynomial features the decision boundary curls around individual points (slides 7 and 21). The fix is the same penalty added to the log loss (slide 21):
Its gradient has exactly the same form as for linear regression, because the log loss's derivative works out to (the week 3 result). Only is different. So the update is the same shrink-then-step rule:
The professor leaves the derivation as an exercise ("this is the calculation you can go through by yourself", 22:44). Problem 2 at the end walks through it.
C is .Slide 23 (the "advanced optimization" cost function) writes gradient(2) as
and likewise for . It must be , as on slide 22. Its also reads "", which should be . (gradient(1), for , correctly has no term.)
Géron notes that scikit-learn's LogisticRegression controls regularization with C, the inverse of the strength: higher C = less regularization, the same direction as the SVM's C in week 5. It also applies an L2 penalty by default.
Why does the professor say "the only thing that changes" for logistic regression is ?
Because the gradient of the log loss, , has the same form as the squared-error gradient, and the penalty's gradient doesn't depend on the loss at all. Swap in the sigmoid for and the update rule is identical.
Choosing λ
is a hyperparameter, so you can't pick it by minimizing training error: that always says . Use held-out data, exactly as in week 2. "How to define the lambda parameter: we can use cross-validation to select lambda" (29:53).
- Pick a grid on a log scale, for example .
- For each , train on the training folds and measure error on the validation fold (or a single validation set). Measure it without the penalty term: the penalty is only for training (Géron).
- Pick the with the lowest validation error, retrain on all training data with it, and report test error once.
The penalty treats every the same, but a feature measured in large units only needs a small weight, so it barely gets penalized, while a feature in small units needs a large weight and gets crushed. Standardize features (mean 0, standard deviation 1) before ridge or lasso. Géron: "It is important to scale the data … before performing Ridge Regression … This is true of most regularized models."
Your validation error is lowest at the largest λ in your grid. What should you do?
Extend the grid to larger values. The true minimum may be beyond the edge you searched. The same applies at the small end.
Regularization in deep learning
Neural networks come later in the course, but the professor previews four regularizers. Networks have huge numbers of parameters, often more than training examples, so they need all the help they can get.
1. L2 penalty (weight decay)
Slide 25 adds the same penalty, summed over every layer's weight matrix :
and backpropagation gets one extra term per layer:
It's the same weight decay as Section 3.
Slide 25 says to stack each weight matrix into one vector "and then we apply sqrt(w1² + w2² …)". The penalty is the squared norm, , with no square root (that's the squared Frobenius norm). The slide's own gradient is the derivative of ; with the square root, it would be different.
The lecture says here is usually very small, around to (23:03), and that it helps when training blows up to NaN. Treat the exact value as problem-dependent: Géron's Keras example uses 0.01. Tune it like any hyperparameter.
2. Dropout
The correct description (Géron Ch. 11, slide 26): at every training step, each neuron (in the chosen layers, never the output layer) is independently "dropped" with probability , the dropout rate. A dropped neuron outputs 0 for that step, for both the forward pass and the backward pass, so the weights attached to it get no gradient that step. The next step draws a fresh random set. At test time nothing is dropped.
Why it works. No neuron can rely on a particular partner being present, so each has to be useful on its own, and no single weight can dominate (the slide's handwritten note: "not a single weight dominates"). Géron's analogy: a company whose employees flip a coin each morning to decide whether to come in would have to spread every critical skill across several people.
A layer outputs . Dropout rate , so the keep probability is . This step's random mask is .
Inverted dropout (slide 26's "most common technique") multiplies by the mask and divides by the keep probability during training: .
Why divide? Each unit survives half the time, so on average its output is , the same as without dropout. That keeps the next layer's input on the same scale at train and test time, so at test time you just use the network as is.
The lecture describes dropout as choosing some weights to leave "unchanged" (not updated) during the forward pass, then choosing a different random set for backpropagation. That isn't dropout. The slide itself says the random dropout vector "is used for forward and back propagation and is the same for them, but it is different for each iteration". Dropout zeroes neurons' outputs (not individual weights; dropping individual weights is a different method called DropConnect), and the same mask is used for the forward and backward pass of a step. Also, slide 26 says the probability is "≪ 1"; typical dropout rates are 10% to 50% (Géron), so not tiny.
3. Data augmentation
The first fix for overfitting is more data. When you can't collect more, make more from what you have: for images, shifted, rotated, flipped or rescaled copies are new training examples with the same label (26:13–27:05). Slide 27 shows Bishop's illustration: the same degree-9 polynomial that overfits 15 points is smooth with 100 points. Only use transformations that don't change the label (a flipped "6" is a "9").
4. Early stopping
Train with gradient descent and watch the validation error after every epoch. Training error keeps falling, but validation error falls, bottoms out and then climbs as the model starts fitting noise. Stop at the validation minimum, even though training error is still decreasing (27:05–28:04). Géron quotes Geoffrey Hinton calling it a "beautiful free lunch".
- In practice, save the weights each time validation error hits a new low, keep training for a while ("patience"), then roll back to the best saved weights. With mini-batch training the curve is noisy, so you can't stop at the first uptick (Géron).
- Validation data, not k-fold. The lecture says the stopping point is usually found "using cross-validation" (28:04). In practice it's a held-out validation set monitored during training. You can use k-fold to pick the number of epochs, but that means training k models.
- Why it's regularization. Starting from small weights, gradient descent grows them gradually, so stopping early keeps them small. For a quadratic loss, Bishop (§5.5.2) shows early stopping behaves like weight decay, with (iterations × learning rate) playing the role of . That's the "probabilistic" explanation the professor alludes to at 28:16.
In the early-stopping figure, what would plain "train until the training loss stops improving" have given you?
The epoch-300 model: training RMSE 0.07 but validation RMSE 2.85, worse than the 2.12 available at epoch 37. Training loss alone can't tell you when you've started overfitting.
Cheat sheet
| Regularized cost | loss (data) + · regularizer (model only) |
| Ridge / L2 | ; never penalize |
| Lasso / L1 | penalty ; sparse weights, feature selection |
| Elastic net | |
| λ extremes | : plain fit (overfit risk); : flat line (underfit) |
| GD (weight decay) | for ; same for logistic |
| Normal equation | , ; invertible for any |
| One feature | ridge ; lasso ; |
| Geometry | larger λ ↔ smaller region; L1 diamond → corners → zeros; L2 disk → no zeros |
| Choosing λ | log grid + validation / cross-validation; scale features first |
| Library knobs | Ridge(alpha): alpha = λ; LogisticRegression(C): C = 1/λ |
| Deep nets | ; dropout (train only, inverted: divide by keep prob); data augmentation; early stopping at validation minimum |
Practice
Q1Why does the regularization sum start at instead of ?
Q2Regularized linear regression with (slide 14) produces:
Q3Gradient descent with , , . Each step multiplies () by what before the usual gradient step?
Q4One feature, centered data with and . The least-squares slope is 1.5. What is the ridge slope with ?
Q5Same data (, ) with the L1 cost . What is the smallest that makes the lasso slope exactly 0?
Q6Geometrically, why does L1 regularization give sparse solutions?
Q7You have examples and features. Which is true?
Q8For regularized logistic regression, is (writing for , and the sum over examples):
Q9You scan . Training errors: 0.10, 0.14, 0.22, 0.40. Validation errors: 0.52, 0.31, 0.29, 0.43. Which do you pick?
Q10Inverted dropout with dropout rate . A unit that survives this training step would have output 2.0. What does it pass on?
Problem 1: For , , find (a) the least-squares line, (b) the ridge line with , (c) the lasso line with , and (d) the smallest for which the lasso slope is 0. (Use the slides' scaling; the intercept is not penalized.)
Means: , . Centered : ; centered : .
. .
(a) , . Line .
(b) , . Line .
(c) , . Line .
(d) The slope is 0 once , so . The line is then . Ridge would need to get there.
Checked numerically: the regularized normal equation gives (1.4, 0.4); a direct minimization of the L1 cost gives (1.55, 0.3) at λ = 5 and slope 0 at λ = 8.
Problem 2: Derive the gradient-descent update for L2-regularized logistic regression (the "calculation you can go through by yourself", 22:44), and show it has the shrink-then-step form.
Let and . A useful fact:
One example's loss is . By the chain rule,
Average over the examples and add the derivative of , which is :
Then regroups as
For there is no penalty, so its update has no shrink factor. This is identical to the linear-regression rule; only differs.