A perceptron (week 3) computes one weighted sum and thresholds it, so its decision boundary is always a straight line. A multilayer perceptron (MLP) feeds the inputs through one or more hidden layers. Each hidden unit computes a weighted sum and passes it through a nonlinear activation function, and the output layer works on those new features instead of the raw inputs. That is enough to solve XOR and, with enough hidden units, to approximate almost any function. The price is training: the loss is no longer convex, and you need the gradient with respect to weights buried deep inside the network. Backpropagation computes all of those gradients with one forward pass and one backward pass, and stochastic gradient descent then uses them exactly as in week 3.
After this you canFrom perceptron to MLP
This lecture opens the deep-learning part of the course. The professor first makes a point about where deep learning fits: it is not a fourth category next to supervised, unsupervised and reinforcement learning. It is a set of tools (neural networks trained by gradient descent) that can be used inside any of those categories (01:28). An MLP for classification, as in this lecture, is plain supervised learning.
The history on slide 2 is short. A single perceptron can only draw a straight-line boundary, so it cannot learn XOR (week 3). Interest in neural networks faded. In the 1980s, the backpropagation algorithm (Rumelhart, Hinton and Williams, 1986) gave a practical way to train networks with hidden layers, and those networks can draw nonlinear boundaries. The professor stresses that backpropagation is still how essentially every deep network is trained today (02:28).
Why the MLP matters (04:02): most learning problems map an input vector to an output vector. A layer that multiplies by a weight matrix and applies a nonlinearity is the basic building block for that mapping, and bigger architectures (CNNs, RNNs later in the course) reuse it everywhere.
The lecture says that the key difference between a perceptron and an MLP "is what's the output": one output node versus several, and "this is the only difference". That is not the important difference. A single layer with several output units (Géron's multi-output perceptron, Figure 10-5) is still a set of linear classifiers and still can't solve XOR. What makes an MLP more powerful is the hidden layer with a nonlinear activation between input and output. Slide 2 says this correctly: the MLP "can make non-linear decision boundary". Several outputs are only needed for multiclass problems (Section 5).
Quick check: an MLP and logistic regression are both "neural networks". Which one can learn XOR, and why?
Only the MLP. Logistic regression is a single unit: thresholded at 0.5 is the line . An MLP first maps the input through nonlinear hidden units, and XOR becomes linearly separable in that hidden space (Section 4).
Layers and the forward pass
Slide 3's network has three layers of units:
- Input layer: just holds the features . It does no computation.
- Hidden layer: units. Each takes a weighted sum of all inputs and applies an activation function .
- Output layer: units. Each takes a weighted sum of all hidden outputs and applies its own activation (sigmoid for two classes, softmax for several).
Every unit in one layer connects to every unit in the next ("fully connected"), and signals only flow forward, so this is a feedforward network. Each layer except the output also has a bias unit that always outputs 1 ( on slide 3), so the weight from it acts as the bias term.
Counting layers is inconsistent across sources. The slides call this a "3-layer perceptron" (counting units). The lecture also calls it "basically two layers" (03:50), counting the two weight matrices, which is how most modern texts and libraries count. Both mean the same network: one hidden layer.
In matrix form (slides 4–5), with the input vector with a leading 1 for the bias:
- input , length
- hidden weights, . Column is the weight vector of hidden unit ; entry connects input to hidden unit
- the weighted sums ("pre-activations", "net input")
- activation function, applied to each entry separately
- hidden outputs (slides: ); with a 1 prepended they become the input to the next layer (slides: of the output layer, 06:50)
- output weights,
- the network's prediction, length
Slide 5 explains the shapes: a single neuron's weights are a vector the size of its input and it outputs one number; a layer of neurons stacks those vectors as the columns of a matrix and outputs a vector of length . Deeper MLPs just repeat the hidden step: .
Number of trainable weights for an –– network: . For the figure: .
Slide 4 writes with rows and columns , which would be . But the hidden bias unit has no incoming weights (it always outputs 1), so there's no column 0. The correct shape is . Also, only includes a bias if starts with the constant 1, as does on slide 3.
Each hidden unit is its own little logistic-regression classifier that answers one yes/no question about the input ("is it on this side of this line?"). The output layer then makes its decision from the answers. The network learns which questions to ask.
Quick check: a network has 4 inputs, one hidden layer of 10 units and 3 outputs. How many weights, including biases?
Activation functions
Why have at all? Without it, two layers compute
which is a single linear layer with weight matrix . Stacking any number of linear layers gives one linear layer. The nonlinearity is what lets depth add power (slide 5: "we need it to help us draw non-linear decision boundary"; Géron makes the same argument with for two linear functions).
Backpropagation also needs to have a useful derivative. That is why the 1986 paper replaced the perceptron's step function (derivative 0 everywhere it exists) with the sigmoid. The lecture names the three common choices (10:10): sigmoid, tanh and ReLU.
| Name | Range | ||
|---|---|---|---|
| Sigmoid (logistic) | |||
| tanh | |||
| ReLU | 1 if , 0 if |
The sigmoid derivative (slide 12, left) is worth being able to derive:
So once the forward pass has computed , the derivative is just , with no extra exponentials. Every factor in the worked example of Section 8 is this.
Géron's practical advice: ReLU is the default for hidden layers today (fast, no saturation for ); tanh's zero-centered output often speeds up training compared with the sigmoid; the sigmoid survives mainly as the output unit for binary classification. The lecture's example uses sigmoids everywhere because they make the hand calculation clean.
The lecture says softmax is usually "not considered an activation function" and is only used on the last layer (10:02–10:34). The second half is right. Géron does call it the output layer's activation ("use the softmax activation function for the whole output layer"), and Keras sets it with activation="softmax". The difference is that softmax acts on the whole output vector at once, while sigmoid/tanh/ReLU act on each unit separately.
Quick check: a sigmoid unit outputs 0.8. What is its local derivative ?
.
Solving XOR with a hidden layer
Week 3 showed that no line separates XOR: and are positive, and are negative. A hidden layer with two units fixes this. Use the step function if , else 0:
In words: XOR = "at least one is on, but not both". Hidden weights are all 1 with biases −0.5 and −1.5; output weights are +1 and −1 with bias −0.5. This is the network in Géron's Figure 10-6.
| (OR) | (AND) | XOR | ||||
|---|---|---|---|---|---|---|
| (0, 0) | 0 | 0 | 0 | −0.5 | 0 | 0 |
| (0, 1) | 1 | 1 | 0 | 0.5 | 1 | 1 |
| (1, 0) | 1 | 1 | 0 | 0.5 | 1 | 1 |
| (1, 1) | 2 | 1 | 1 | −0.5 | 0 | 0 |
With sigmoids instead of steps (so it could be trained by backprop), scale the weights up so each sigmoid is nearly a step: , , . Computed exactly, the outputs are 0.0001, 0.9999, 0.9999, 0.0001 for the four inputs in table order. Threshold at 0.5 and you get XOR.
A hidden layer is a learned feature map, like the hand-made features of week 3 () or the kernel map of week 5, except the network chooses the features itself during training.
Quick check: in the step network, what does the output become if you change the output bias from −0.5 to −1.5?
is −1.5, −0.5, −0.5, −1.5 for the four inputs, so the output is 0 for every input. The hidden features are still right, but the output line is now on the wrong side of every point.
Multiclass output: one-hot, softmax and the loss
Slide 6 brings back the Iris data: 4 features, 3 species. A perceptron has one output and handles two classes. For classes the MLP gets output units, and the labels are written as one-hot vectors (slide 7):
setosa = (1, 0, 0), virginica = (0, 1, 0), versicolor = (0, 0, 1)
The output layer should produce a probability for each class. The softmax function turns the output layer's scores into positive numbers that sum to 1:
- score (logit) of output unit , i.e. entry of
- predicted probability of class ; predict the class with the largest
Slide 7 writes the denominator as , reusing as both the free index and the summation index. Read it as a sum over a separate index , as above.
To train, the network needs a loss comparing with the one-hot . The lecture connects this to entropy from the decision-tree lecture (10:56): cross-entropy is .
With a one-hot , only the true class survives the sum: . Predict the true class with probability 0.9 and the loss is 0.105; with 0.6 it's 0.511; with 0.1 it's 2.303. Confident mistakes cost a lot.
Scores , true class 1 ().
- Exponentials: , , . Sum = 11.212.
- Softmax: . (Order is kept; the largest score gets the largest share.)
- Loss: .
- Gradient with respect to the scores: Gradient descent raises and lowers the others. (Checked numerically by finite differences.)
That last line is the reason softmax and cross-entropy are always paired: their combined gradient is simply prediction minus target, . The same is true for a sigmoid output with binary cross-entropy: . (Problem 2 derives it.)
MSE as an alternative (slide 9, 12:42): . It has the same form for any number of classes and also works when the outputs are not probabilities, which is why it's the standard loss for regression MLPs (the professor's handwritten note on slide 9). For a regression MLP, Géron puts no activation on the output unit.
Slide 9 says MSE "is always easy to be optimized" and you can freely pair it with a logistic or softmax output. It works, but it isn't the easy choice for classification: with a sigmoid or softmax output, the MSE gradient includes the saturating slope , so learning stalls exactly when the network is most wrong (figure above). This is why Géron's Table 10-2 lists cross-entropy as the loss for every classification setup, and MSE (or MAE/Huber) for regression. Neither loss makes an MLP convex.
Slide 6 numbers the classes setosa = 0, virginica = 1, versicolor = 2. scikit-learn's load_iris() uses setosa = 0, versicolor = 1, virginica = 2. The numbering is arbitrary; just check which one your data uses before reading a confusion matrix.
Quick check: scores . What does softmax give, and what is the cross-entropy if class 2 is true?
All exponentials are equal, so . Loss . This is the loss of a network that knows nothing, a useful sanity check at the start of training.
The chain rule and computation graphs
Training uses stochastic gradient descent (SGD), as in week 3: for each sample or mini-batch, move every weight a small step against its gradient, . For the output weights this is no harder than logistic regression. The new problem (slide 9) is computing for a hidden weight, which affects the loss only through everything after it.
The tool is the chain rule (slide 10, 13:44): for a composition, multiply the derivatives of each step.
Slide 11 writes the whole 2-layer network as exactly such a chain, one step per operation:
This picture is a computation graph (14:46). The professor's tips (16:06–16:32): everything is gradients; every gradient comes from the chain rule; and to evaluate one you plug in the current values of the variables, which is why the forward pass must store them. Working from the loss backward means each partial product is computed once and reused for every weight below it. Géron calls this reverse-mode autodiff: one backward pass gives the gradient with respect to every weight, which is efficient when there are many inputs (weights) and one output (the loss).
Slide 11 writes the loss as . The minus sign is wrong: MSE is , and minimizing the slide's version would increase the error. With the correct sign:
Slide 12's "" is really (it includes , so it's a derivative with respect to , not ), and the last step silently drops . The version used in the worked example (slides 16–18) is the clean one: take the loss for one output, so the 2 cancels, and define . The update then moves downhill. (Mitchell §4.5 uses exactly this convention.)
Quick check: . What is ?
Chain rule: . At 0, , so and .
Backpropagation: the general rules
Write the forward pass one unit at a time, as the slides do on 14: unit receives (the are the outputs of the units feeding it, including the constant bias input ) and outputs . Use the loss over the output units .
Define each unit's error term : how much the loss would drop per unit increase of that unit's weighted sum. Then:
- weight from unit into unit (slides' convention: first index = source)
- output of the source unit (an input, a hidden output, or 1 for a bias)
- sigmoid slope ; for another activation use its
- target for output
- learning rate
Read the hidden rule as "blame flows backward along the same weights the signal flowed forward on". Unit contributed to each next-layer unit through , so it gets times that unit's error, summed, then scaled by its own slope.
Why the hidden rule holds. affects only through the units it feeds. Chain rule:
Negate both sides and you have the formula. The weight rule is one more link: .
Matrix form for one sample, with the slides' convention and Bishop's sign ():
- element-wise product
- without its bias row (the bias unit has no )
The gradient of a weight matrix always has the same shape as the matrix: (layer input) × (layer error)ᵀ. Checking shapes is the fastest way to catch a missing transpose.
The slides (and Mitchell) define , so and the update adds . Bishop (§5.3, eq. 5.54) defines , so (for matched output/loss pairs) and the update subtracts . Both give the identical weight change. Mixing them (Bishop's with the slides' plus sign) climbs the loss. Pick one and stay with it.
Compute every first, with the current weights, and only then update the weights. If you update before computing , uses the wrong weight. The slide example does it in the right order: slide 16 computes all deltas, slides 17–18 update.
Quick check: a hidden unit outputs 0.5 and feeds two output units through weights 2 and −1, whose deltas are 0.1 and 0.3. What is its δ?
Worked example: one forward pass, backward pass and update
"For this part, I would not go through in details. … You can just look at the slides [and] understand how to use the chain rule in backpropagation to get the gradients for the parameters." And: "I will not go through the details for this calculation, but you can just go through the slides. If you have a question on that, let me know." The example was assigned as self-study, so here it is, every number checked.
Setup (slide 13). Two inputs, two sigmoid hidden units , one sigmoid output . is the bias input, always 1. The training sample is with target (one row of logical OR). Loss , learning rate .
| Into | Into | Into |
|---|---|---|
| (bias) | (bias) | (bias) |
The diagrams on slides 13–16 show , but every number on the slides was computed with −3.92: slide 18's table lists −3.92, its update −3.9078 = −3.92 + 0.0122, and slide 19's only comes out with −3.92. (With −3.93, slide 15's would be 0.034, not 0.04.) This guide uses −3.92. Also, the handwritten note on slide 13 labels that edge "w15"; it comes from , so it is .
Forward pass (slides 14–15). Each unit: weighted sum, then sigmoid.
- , so .
- , so .
- so .
- Error ; loss .
Backward pass (slide 16). Deltas from the output down, all with the current weights.
- Output:
- Hidden :
- Hidden :
The slides round to 0.51 before computing , which gives 0.1225 instead of 0.1222. Harmless, but it's why their later digits differ slightly from exact values.
Gradients and update (slides 17–18). and :
| old | (source output) | new | |||
|---|---|---|---|---|---|
| 1 | 0.00432 | 1 | −0.00432 | 1.00043 | |
| 3 | 0.00432 | 1 | −0.00432 | 3.00043 | |
| 4 | 0.00432 | 0 | 0 | 4 (unchanged) | |
| −6 | 0.12219 | 1 | −0.12219 | −5.98778 | |
| 6 | 0.12219 | 1 | −0.12219 | 6.01222 | |
| 5 | 0.12219 | 0 | 0 | 5 (unchanged) | |
| −3.92 | 0.12219 | 1 | −0.12219 | −3.90778 | |
| 2 | 0.12219 | 0.9820 | −0.11999 | 2.01200 | |
| 4 | 0.12219 | 0.5000 | −0.06109 | 4.00611 |
Every entry in the column matches a numerical finite-difference gradient with to six decimal places. That check (called gradient checking) is the standard way to test a hand-written backprop. Note that weights leaving an input that is 0 () don't move: that input contributed nothing to this prediction, so it gets no blame.
Verification (slide 19). Run the forward pass again with the new weights: , ; , ; , . The error dropped from 0.4890 to 0.4761 (loss from 0.1196 to 0.1133). One small step downhill. Training repeats this for every sample, many times over ("update again", says the handwriting on slide 19).
Quick check: why did barely change (by 0.0004) when changed 28 times more?
Both come from input , so the difference is entirely vs . is saturated: its output 0.982 sits on the flat part of the sigmoid, slope , versus 0.25 for . It also feeds the output through a smaller weight (2 vs 4). Saturated sigmoid units learn slowly, which is one reason ReLU became the default.
Training in practice, and the limits of MLPs
You won't do this by hand in practice (17:39). PyTorch and TensorFlow compute all gradients automatically from the computation graph. Your job is to choose:
- the architecture (number of layers, units per layer, activations),
- the loss (cross-entropy for classification, MSE for regression),
- the optimizer (SGD, or Adam, which the lecture mentions at 20:20) and its learning rate,
- the batch size and number of epochs.
One training loop, as Géron summarizes it: take a mini-batch; forward pass, keeping every intermediate value; compute the loss; backward pass to get every gradient; one gradient-descent step; repeat. One pass over the whole training set is an epoch.
Initialize randomly. If all weights start at zero (or any equal values), every hidden unit computes the same thing, gets the same δ, and receives the same update, so they stay identical forever and the layer acts like one unit. Random initialization breaks this symmetry (Géron).
Limitations (slide 20, 18:34–21:10).
- Non-convex optimization. With hidden layers, the loss is "highly non-convex" (18:36): it has many local minima and flat regions. Gradient descent only guarantees a point where the gradient is zero, not the best one, and the result depends on the random initialization and step size (the professor's handwritten notes on slide 20). Contrast with the SVM (week 5), whose training problem is convex. The professor mentions research showing that two-layer ReLU networks can reach a global optimum under certain assumptions (19:16), but there's no general guarantee. His summary: "how can we guarantee this local minimum is good? This is the question."
- Black box. With many parameters mapping into a high-dimensional hidden space, it's hard to see why a trained network makes a given decision (19:42).
He adds that optimization is "still a research topic but not so hard" in practice: modern optimizers usually find good solutions.
"Remember in the homework one, we have the implementation of perceptrons … you can define the perceptron as single layer MLP. … You can optimize the perceptron using stochastic gradient descent to learn parameters. Try it. … see the difference between perceptron and the multilayer perceptron." Also (21:54): "understand how to use MLP for your binary, for your multi-class classification problem." Expect to set up an MLP classifier (one-hot targets, softmax output, cross-entropy loss, SGD) and to compare it with the HW1 perceptron.
Quick check: you train the same MLP twice on the same data and get different test accuracies. Is something broken?
Not necessarily. Different random initializations (and different mini-batch orders) can lead gradient descent to different local minima of a non-convex loss. Common practice is to fix the random seed for reproducibility, or to train several times and keep the best on validation data.
Cheat sheet
| Layer | (with ), ; is (inputs+1) × units |
| Weight count | –– net: |
| Why nonlinear | linear layers compose to one linear layer: |
| Sigmoid | , |
| tanh, ReLU | ; ReLU , slope 0 or 1; ReLU is the default hidden activation |
| XOR net | , , |
| Softmax | ; one output per class, one-hot targets |
| Cross-entropy | ; binary: |
| Output gradient | softmax + CE or sigmoid + BCE: |
| Loss choice | classification: cross-entropy; regression: MSE (no output activation) |
| Output δ (slides) | for |
| Hidden δ | , using the old weights |
| Update | (); zero input → no change |
| Matrix form | , same shape as |
| Gradient check | |
| Practice | random init (zeros → identical units); mini-batch SGD/Adam; non-convex → local minima, seed-dependent |
Practice
Q1You build a 3-layer network but forget the activation functions (every unit is linear). On XOR it will:
Q2An MLP has 4 inputs, 5 hidden units and 3 outputs, with biases. How many weights?
Q3In the XOR network , , , what are for input ?
Q4A sigmoid unit's output is 0.9. Its derivative is:
Q5Output scores are . What is the softmax probability of class 1?
Q6Softmax outputs and the true class is class 2. The cross-entropy loss is:
Q7With a softmax output and cross-entropy loss, the gradient of the loss with respect to the output scores is:
Q8In the lecture example . After one backprop step, the weights and (from ):
Q9You initialize every weight of an MLP to 0.5. What goes wrong?
Q10Which statement about training an MLP with gradient descent is true?
Problem 1: Same network and weights as Section 8, but train on the OR row , target , . Do the forward pass, all three δ's, and the updates. Which weights change?
Forward. , . , .
Loss . The network already says "probably 0".
Backward.
Updates . Inputs , so don't change. The rest:
; ; ; (source ); (source , essentially unchanged).
The new output is 0.07951, slightly closer to 0. All the deltas are negative because the output should go down, so the biases and decrease.
Checked in Python, including a finite-difference check of every gradient. Side note: on the other OR rows this network currently outputs 0.298 for (0,1) and 0.886 for (1,1), so it still needs many more updates.
Problem 2: Show that for softmax and cross-entropy with a one-hot , .
Take logs first: . So
using . Differentiate with respect to :
Check with Section 5's numbers: , gives , which matches the finite-difference gradient.
This is why the output δ in a softmax classifier is just "prediction minus target", with no factor to shrink it. The binary case (sigmoid + binary cross-entropy) works the same way and gives .