The video covers two decks. Lecture 3a (classification) is borrowed from a UIUC course and lecture 3b (regression) from a deep-learning course, so the notation shifts between them. This guide puts them in the order that makes sense: first linear regression, fit exactly with a formula or step by step with gradient descent; then linear classification with the perceptron, which learns by nudging a line every time it makes a mistake; then what happens when you make the perceptron smooth enough for gradient descent, which is where logistic regression comes from.
After this you canOne formula, two jobs
- weights: how much each feature matters, and in which direction
- bias (offset, intercept)
| Task | Output | Example |
|---|---|---|
| Regression | the number itself | predict a house price |
| Classification | its sign: | dog (+1) or cat (−1) |
Append a constant 1 to every input: and . Then , and the bias is just another weight. Both decks do this. It's why 3a writes with no , and why 3b's matrix has a column of 1s.
Least-squares linear regression
Pick the line whose predictions have the smallest total squared error on the training data (the MSE from week 2, without the ):
Setting both partial derivatives to zero gives a closed-form answer (slide 3):
Data: , , , . So , , .
Line: . Predictions 1.9, 3.3, 4.7, 6.1; misses 0.1, −0.3, 0.3, −0.1; SSE = 0.2.
Many features: the normal equation
Stack the inputs as rows of a matrix (with the column of 1s from the bias trick) and the targets as a vector . Minimizing by setting its gradient to zero gives:
Check with the example: , , and solving gives , the same answer.
The formula needs to be invertible. That fails if one feature is a combination of others (e.g. height in cm and height in inches) or if there are fewer samples than weights. Inverting is also slow for many features (roughly cubic in ). Both are reasons to use gradient descent instead (Géron Ch. 4 covers this trade-off).
"Linear" in what?
"Linear regression" means linear in the weights, not necessarily in the inputs. You can feed in , or as extra features and it's still linear regression, solved with the exact same formula:
| Model | Linear regression? |
|---|---|
| yes | |
| yes: each just multiplies a feature | |
| yes | |
| yes for (this is slide 5's "log-linear regression") | |
| no: the s sit inside and |
Slide 6 lists under "not linear regression models", while slide 5 presents the same thing as "log-linear regression". Both are defensible. It isn't linear for , but it is an ordinary linear regression on the transformed target . On an exam, say exactly that.
In the lecture, polynomial regression is grouped with "nonlinear functions" that "cannot have a closed-form solution", so you'd need gradient descent. That's only true for models that are nonlinear in the weights (the last row of the table). A polynomial in is still linear in its weights, so the normal equation solves it exactly. Géron does precisely this in "Polynomial Regression": add as features, then fit ordinary linear regression.
Fitting by gradient descent
Instead of solving exactly, start from and repeatedly step downhill on the loss. (For the step-by-step mechanics see the sample guide on gradient descent.) The loss for linear regression is a convex bowl, so there's one minimum and gradient descent finds it.
Calculus refresher (3b slides 11–32): derivative, chain rule, gradient
Derivative = slope: . E.g. .
Chain rule: . E.g. . The chain rule is all of backpropagation later in the course.
Gradient = vector of partial derivatives. For : , which at is .
Multivariable chain rule: . Slide 29's example (, , ) gives , which matches differentiating directly.
For one training point with loss and , the chain rule gives:
- the error: positive if the prediction was too low, so the weights move to raise it
- learning rate
The "vectorized" column writes and then updates . Taken literally, that's , which moves uphill. The gradient is missing a minus sign. The "for-loop" column on slide 8 and slide 9 have it right: . The final rule above is the correct one. (The on those slides is the identity function for linear regression; slide 36 says so.)
Start , . Training point . Prediction , error .
, . The new prediction at is : closer to 5, as it should be.
Updating after every single point is online (stochastic) mode. Updating once per pass with the summed gradient is batch mode. Updating on small groups is mini-batch (slide 38), the usual choice in deep learning: it vectorizes well and is less noisy than one point at a time.
In the lecture, "batch mode" is used loosely for mini-batch (e.g. 10–20 images per update), and the professor points out that papers saying "SGD" almost always mean mini-batch too. Strictly: batch = the whole training set per update, stochastic = one sample, mini-batch = a small random subset. The gradient formula is the same in all three; only which data you sum over changes. A constant factor like can be folded into .
"This is basically the homework you can take: try to think about how to use gradient descent to solve this least-squares regression, no matter if it's linear functions or nonlinear functions." The recipe: write the loss, take its gradient with respect to every weight (chain rule), then loop . Check your linear-regression answer against the normal equation; they should agree.
Why gradient descent is trustworthy here: least-squares loss is convex, so with a small enough it reaches the global minimum. For non-convex losses (later, neural networks) it can get stuck in a local minimum. The professor recommends Boyd & Vandenberghe, Convex Optimization (it's in the textbook folder; Ch. 9 covers gradient descent) for more.
Scale your features
Classifying with a line
The lecture opens with "tell dogs from cats". One feature alone fails: weight (a Maine Coon cat outweighs a Chihuahua) or "comes when called" (some dogs ignore strangers). Combining them works better: with = times it comes when called (out of 40) and = weight in pounds, say "dog if ". The boundary is a straight line, which is why this is called a linear classifier.
With more than two classes, give each class its own weights and pick the class with the highest score: . The plane splits into regions with straight-edged borders (slide 31).
The perceptron
Rosenblatt's perceptron (1958–59) was modeled on a neuron: weighted inputs are summed and the output is a binary "fire or don't". Its big idea was that it learns from examples:
- Start with any weights (random or zero). Labels are .
- For each training point, predict (bias trick included).
- If : do nothing. If wrong: .
- Repeat passes (epochs) until a full pass makes no mistakes.
Two conventions from the lecture: the professor defines (a score of exactly zero counts as positive), and in practice you stop after a fixed number of epochs (full passes over the data), whether or not it has converged. He also notes that is "large in practice".
"In the homework we have the implementation of this perceptron algorithm." When you code it: append the constant 1 to every , use labels ±1 (not 0/1), loop over epochs and then over samples, update only on mistakes, and stop when an epoch has zero mistakes or the epoch limit is hit. Scaling the features first (see below) makes it behave far better.
Why does adding help? After the update, the score for that point changes by , which moves it in the direction of the correct sign.
= [comes when called /40, weight in lb, 1]. Start with : "dog if it comes more than 20 times".
| Animal | x | True y | Score wᵀx | Prediction | New w |
|---|---|---|---|---|---|
| Presa Canario | [1, 100, 1] | dog +1 | 1 − 20 = −19 | cat ✗ | [1,0,−20] + [1,100,1] = [2, 100, −19] |
| Maltese | [40, 10, 1] | dog +1 | 80 + 1000 − 19 = 1061 | dog ✓ | unchanged |
| Maine Coon | [0, 20, 1] | cat −1 | 0 + 2000 − 19 = 1981 | dog ✗ | [2,100,−19] − [0,20,1] = [2, 80, −20] |
The slides stop there, but it's worth checking the result. With , the Maine Coon scores : still classified as a dog. The boundary means "dog if it weighs more than about a quarter pound". Every real cat is now a "dog".
When does it work? Separability and XOR
A dataset is linearly separable if some line (plane, hyperplane) puts every positive on one side and every negative on the other.
- If the data is separable, the perceptron is guaranteed to find a separating line in a finite number of mistakes, even with . The proof idea (slides 22–26): replace each by , which flips the negatives so that "correct" means for every point. Each mistake adds that point to , turning toward the side where all the flipped points lie.
- If it isn't separable, the perceptron never settles: some point is always wrong, so keeps getting pushed around. The fix is a decaying learning rate, e.g. for the th update, so the wiggling dies down (slide 27).
This is the famous limitation from Minsky & Papert's 1969 book Perceptrons. Many people read it as "neural networks can't learn XOR" and the field cooled for years, even though the same book showed that a two-layer network handles XOR easily. You'll see that in the multilayer perceptron (week 13).
Making it differentiable: toward logistic regression
The perceptron's update rule looks ad hoc. Why not just do gradient descent? Because is a step function: its derivative is zero everywhere except at the jump, so the gradient gives no direction. The fix is to swap the step for a smooth S-curve.
"Logistic regression here is a classifier. It's not a regressor, it's not linear regression." The name is historical: it regresses a probability, then thresholds it to pick a class.
The slides' version (3a, slides 36–44): output , squared-error loss , and using :
Compare it with the perceptron's . Same idea, but the step is scaled by how wrong the output is, , and by how steep tanh is there, . Points the model is already sure about barely move it.
The slides call this "logistic regression". Géron (Ch. 4, Eq. 4-13 to 4-18) and most references define logistic regression differently: labels , output read as a probability, and the log loss instead of squared error. That gives the clean gradient , and log loss is convex, while squared error on top of an S-curve is not. Know both. If an exam just says "logistic regression", the textbook version is the safe default.
| Perceptron | Slides' "differentiable perceptron" | Logistic regression (Géron) | |
|---|---|---|---|
| Labels | ±1 | ±1 | 0 / 1 |
| Output | = P(class 1) | ||
| Loss | (none; mistake-driven) | squared error | log loss (cross-entropy) |
| Update | on mistakes only | ||
| Who updates | misclassified points | every point, a little | every point, a little |
One more correction: slide 44 says that for logistic regression, "if then do nothing". With tanh, is never exactly ±1, so in practice every point contributes a small update. That's the real difference from the perceptron, as the last row shows.
Cheat sheet
| Linear model | ; bias trick: append 1 to |
| Least squares (1-D) | , |
| Normal equation | ; needs invertible |
| "Linear" | linear in the weights; , features are fine |
| GD update (squared loss) | , |
| Online / batch / mini-batch | update per point / per full pass / per small group |
| Feature scaling | standardize → round loss bowl → faster GD and saner perceptron updates |
| Linear classifier | ; ⟂ boundary, points to + side |
| Perceptron | on a mistake: ; converges iff data separable |
| Multi-class | |
| XOR | not linearly separable; needs two layers |
| tanh / sigmoid | ; ; |
| Logistic regression | , log loss, gradient , convex, no closed form |
Practice
Q1Why do the slides append a 1 to every feature vector?
Q2Perceptron with sees , . What happens?
Q3Perceptron (η = 1) with sees , . New w?
Q4You train a perceptron with η = 1 on the four XOR points. What happens?
Q5Is a linear regression model?
Q6Your features include temperature in °C and the same temperature in °F. What breaks in the normal equation?
Q7For squared loss , which is the correct gradient-descent update?
Q8Gradient descent zig-zags and converges slowly. One feature ranges 0–1, another 0–10,000. Best fix?
Q9What are tanh(0) and the range of tanh?
Q10Which statement about (textbook) logistic regression is true?
Problem 1: Fit a least-squares line to (0, 1), (1, 3), (2, 4) by hand.
, , , .
Numerator: . Denominator: . So .
.
Check: predictions 1.167, 2.667, 4.167; misses −0.167, 0.333, −0.167, which sum to zero, as least-squares misses always do when there's a bias term.
Problem 2: Prove no line can classify XOR (positive at (0,1) and (1,0), negative at (0,0) and (1,1)).
Suppose one did. The four points require:
(0,0) negative: . (1,1) negative: .
(0,1) positive: . (1,0) positive: .
Add the two positive conditions: , so . That contradicts . So no such line exists. ∎