Every classifier so far (logistic regression, k-NN, SVM) learned the answer to "given these features, which class?" directly. This week flips the question around: learn what each class looks like, meaning how likely each feature value is within that class, and how common each class is. Then Bayes' rule turns those two ingredients into the probability of each class for a new example. That is a generative classifier. Modeling all features jointly needs far too much data, so naive Bayes assumes the features are independent once the class is known. With that assumption, training is just counting, and prediction is multiplying a handful of numbers from a table. The professor calls this the last supervised-learning method of the course, and repeats one formula as the thing to remember: posterior ∝ likelihood × prior.
After this you canProbability toolkit
The professor assumes you've taken a probability course and runs through the vocabulary quickly. Everything in this lecture is built from five facts about two random variables and :
| Name | Notation | Meaning |
|---|---|---|
| Prior (marginal) | probability of on its own, before seeing anything else | |
| Conditional | probability of once you know | |
| Joint | probability that both happen | |
| Product rule | a joint splits into conditional × marginal, in either order | |
| Independence | equivalently : knowing one tells you nothing about the other |
One more fact appears constantly: the sum rule (marginalization). To get the probability of alone, add the joint over every value of :
The professor skipped this one ("if you like, you can go through by yourself", 28:53). Roll two fair dice. A = die 1 shows 3, B = die 2 shows 1, C = the dice sum to 8. All 36 outcomes are equally likely.
- and .
- : the sums of 8 are (2,6), (3,5), (4,4), (5,3), (6,2).
- . The dice don't influence each other, so knowing die 2 changes nothing.
- . Given die 1 is 3, you need die 2 to be 5, one chance in six.
- , which equals : A and B are independent.
- , only the outcome (3,5). But . So A and C are not independent: learning die 1 is 3 raises the chance of a sum of 8 from 5/36 to 6/36.
Quick check: is B (die 2 shows 1) independent of C (sum is 8)?
No. If die 2 shows 1, a sum of 8 needs die 1 to show 7, which is impossible. So , but .
Bayes' rule: posterior ∝ likelihood × prior
The product rule lets you write the joint of features and class in two orders:
Set the two equal and divide by . That's all Bayes' rule is:
- prior: how common each class is before looking at the features (slide 15: a common cold is far more likely than avian flu)
- likelihood or class-conditional model: how plausible these features are if the class is ("assuming it's avian flu, do the symptoms look plausible?")
- evidence or normalization constant: the same for every class, computed with the sum rule
- posterior: what we actually want, the probability of each class given the features
To classify, pick the class with the largest posterior. This is the MAP decision rule (maximum a posteriori; slides 14 and 21). The evidence is the same number for every class, so it can't change which class wins, and we can drop it:
If you knew the true distributions, this rule would be the best possible classifier, the Bayes optimal classifier (Shalev-Shwartz & Ben-David §24.2). Everything else this week is about estimating and from data.
The professor says this three times: "This is the key for the lectures"; "the key information from this class is remember for naive Bayes classifier, the posterior will be proportional to the likelihood times prior"; and in the summary, "you need to remember the Bayes rule, which is the posterior is proportional to the multiplication between likelihood and the prior." Know cold.
Dropping is fine for choosing a class, but the numbers don't add up to 1. To report an actual probability, divide each by their sum. Section 7 shows this with the slides' own numbers.
Worked example: which machine made the defect?
A factory's output comes from three machines: A makes 20%, B 30%, C 50%. Their defect rates are 5%, 3% and 1%. An item picked at random turns out to be defective (call that event D). What's the chance machine C made it?
- Priors (share of output): , , .
- Likelihoods (defect rate of each machine): , , .
- Evidence by the sum rule: So 2.4% of all output is defective.
- Posterior: Likewise and .
"At random, the term is very important." Bayes' rule with these priors only works because the item was drawn at random from the total output, so the share of output really is the prior probability of each machine. If someone hand-picked items from machine C's bin, the prior would be different.
Slide 5 calls the machines and the defect , the reverse of the classification convention (where is the class we infer). The math is unaffected; just don't let the letters confuse you. Here the "class" is the machine and the "feature" is the defect.
Quick check: what's the probability that a non-defective item came from machine C?
, so the numerator is . The evidence is . So , barely above the prior of 0.5, because a good item is the common case for every machine.
Generative vs discriminative classifiers
Bayes' rule has two sides, and the professor's annotation on slide 3 labels them: the left side is the "discriminative model", the right side the "generative model".
- Discriminative classifiers model directly. Pick a functional form (for logistic regression, a sigmoid of a linear score) and fit its parameters to predict labels. They never say what the data looks like. Slide 12 draws this as one box that takes and outputs .
- Generative classifiers model and : one probability model per class describing how that class's data is distributed (slide 13 draws boxes, one per class), plus the class frequencies. Bayes' rule then produces . The name comes from the fact that such a model could generate new fake data: pick a class from , then sample features from . Slide 9's arrows show this: for generative, for discriminative.
Slide 6 says generative classifiers "assume some functional form for P(X|Y), P(X)". The professor corrects it live: "this notation wrong ... this part should be P(Y)." A generative classifier learns and the class prior . Slide 9 has it right.
Slide 7 writes . With that is , which decreases as grows, contradicting the same slide's when . The standard form (week 3) has a minus sign: .
Which is better? The lecture's answer (25:02–27:25 and 55:19): with limited data, discriminative models usually win; with very large data and a generative model that fits the data well, generative can win, because modeling the full distribution captures more. Generative models are also easier to interpret, and they can handle missing features gracefully (Section 12). Their weakness is that you must assume a form for , such as Gaussian, and if the data doesn't follow it the model is biased no matter how much data you have.
Shalev-Shwartz & Ben-David (Ch. 24 intro) argue the other way in general: the discriminative approach "directly optimizes the quantity of interest", and learning the full distribution is usually harder than learning a good predictor (Vapnik's principle: don't solve a more general problem as an intermediate step). The classic empirical comparison, naive Bayes vs logistic regression (Ng & Jordan, 2001), found that naive Bayes often does better with few examples because it converges faster, while logistic regression tends to win as the data grows. The professor's claim holds only when the generative model's assumptions really are correct. On an exam, the safe statement is: generative models need correct distributional assumptions; when those assumptions are wrong, discriminative models usually reach lower error with enough data.
Quick check: is k-NN generative or discriminative? What about an SVM?
Both are discriminative, in the broad sense: neither models how the features are distributed within each class. An SVM doesn't even produce probabilities; it learns the boundary directly. Naive Bayes is the course's main generative classifier.
MLE vs MAP
The professor spends several minutes hand-writing two estimation principles on the slides. They come up again in training (slide 17: "maximum likelihood") and smoothing (slide 18: "MAP estimation instead of MLE"), so it's worth getting them straight. Let be the model's parameters (for naive Bayes, the probabilities in the tables) and the training data.
- MLE
- maximum likelihood estimation: the parameters under which the observed data is most probable
- MAP
- maximum a posteriori estimation: the most probable parameters given the data, which means likelihood times a prior on the parameters
You see 0 heads in 5 flips. The MLE of is the observed fraction, (Shalev-Shwartz & Ben-David derive this in §24.1 by setting the derivative of the log-likelihood to zero). That says heads is impossible, which is absurd after five flips.
MAP with a prior that believes the coin is roughly fair, a Beta(2, 2) prior, gives . The prior acts like one extra head and one extra tail. That's exactly the "+1 / +2" smoothing on slide 18 (Section 9).
In lecture, MLE is described as maximizing "without the prior", and MAP as maximizing , i.e. using the class prior. That describes the two decision rules: pick the class with the largest likelihood (the "ML" rule, which pretends all classes are equally common) vs the largest posterior (the MAP rule from Section 2). But slides 17–18 use the words for parameter estimation, where the prior is , a prior on the table entries, not the class prior . Note that slide 17's naive Bayes training estimates the class prior and is still called MLE. Keep the two senses separate: naive Bayes always classifies with the MAP rule (it uses ), and it can estimate its tables either by MLE (plain counts) or MAP (smoothed counts).
Quick check: with a uniform prior on θ, how do MAP and MLE compare?
They're identical: a uniform is a constant, so it doesn't move the argmax. MAP only differs from MLE when the prior prefers some parameter values over others.
The naive assumption
Bayes' rule needs , the probability of the whole combination of feature values within a class. Estimating that by counting is hopeless: with binary features there are combinations per class, and most will never appear in your training set. Shalev-Shwartz & Ben-David put it as "the number of examples we need grows exponentially with the number of features."
The naive Bayes assumption (slide 16): all features are independent given the class label. Then the joint likelihood factors into one small term per feature:
The payoff. For binary features and two classes, the full model needs numbers; naive Bayes needs (Shalev-Shwartz & Ben-David §24.2). With , that's over two billion versus 61. For play tennis (attributes with 3, 3, 2, 2 values), the full table has cells per class; naive Bayes uses probabilities per class.
In lecture the assumption is phrased as "each data is independent from others." That sounds like the training examples are independent of each other (the usual i.i.d. assumption, a different thing). The naive Bayes assumption is about the features within one example, and it is conditional on the class: . Slide 16 states it correctly ("independent given the class label Y").
Features can be strongly correlated overall and still conditionally independent. Example: in a population of children and adults, height and weight are highly correlated, mostly because both depend on age group. Within adults alone, they're much less related. Naive Bayes only needs the within-class version (slide 26's height/weight example makes exactly this assumption).
For most real data the assumption is false: in text, "San" and "Francisco" are not independent. Yet naive Bayes "works surprisingly well anyway" (slide 22). Classification only needs the right class to have the highest score, not accurate probabilities. Double-counting correlated evidence tends to make the probabilities overconfident (pushed toward 0 or 1) without changing which class wins.
Training by counting: the play-tennis example
Training a discrete naive Bayes model is "easy" (slide 17): every probability is a fraction of records. These are the MLE estimates.
The data is the same 14 days used for decision trees (Mitchell's Table 3.2, slide 19):
| Day | Outlook | Temp. | Humidity | Wind | Play |
|---|---|---|---|---|---|
| D1 | Sunny | Hot | High | Weak | No |
| D2 | Sunny | Hot | High | Strong | No |
| D3 | Overcast | Hot | High | Weak | Yes |
| D4 | Rain | Mild | High | Weak | Yes |
| D5 | Rain | Cool | Normal | Weak | Yes |
| D6 | Rain | Cool | Normal | Strong | No |
| D7 | Overcast | Cool | Normal | Strong | Yes |
| D8 | Sunny | Mild | High | Weak | No |
| D9 | Sunny | Cool | Normal | Weak | Yes |
| D10 | Rain | Mild | Normal | Weak | Yes |
| D11 | Sunny | Mild | Normal | Strong | Yes |
| D12 | Overcast | Mild | High | Strong | Yes |
| D13 | Overcast | Hot | Normal | Weak | Yes |
| D14 | Rain | Mild | High | Strong | No |
Training. There are 9 Yes days and 5 No days, so and . For each attribute value, count how often it occurs among the Yes days and among the No days. For example, 2 of the 9 Yes days are sunny, so (the professor works this one at 38:30–39:33). The full set of tables (slide 20):
| Attribute | Value | P(value | Yes) | P(value | No) |
|---|---|---|---|
| Outlook | Sunny | 2/9 | 3/5 |
| Overcast | 4/9 | 0/5 | |
| Rain | 3/9 | 2/5 | |
| Temperature | Hot | 2/9 | 2/5 |
| Mild | 4/9 | 2/5 | |
| Cool | 3/9 | 1/5 | |
| Humidity | High | 3/9 | 4/5 |
| Normal | 6/9 | 1/5 | |
| Wind | Strong | 3/9 | 3/5 |
| Weak | 6/9 | 2/5 |
Testing. Classify the new day (slide 21). Look up one number per attribute in each column and multiply by the prior:
- Yes score: .
- No score: .
- Decide: No has the larger score, so predict No. That's the MAP rule.
- Normalize to get real probabilities. The evidence is the sum of the two scores, 0.02586:
Slide 21 writes "P(Yes|x') = 0.0053" and "P(No|x') = 0.0206". Those are , not posteriors (they add to 0.026, not 1). The professor points this out at 42:42: "these two values are not normalized ... they do not divide by P(X)." The true posteriors are 0.205 and 0.795, matching Mitchell §6.9.1. The decision is the same either way.
Quick check: what does naive Bayes predict for (Sunny, Hot, Normal, Weak)?
Yes: . No: . Predict Yes, with
Normal humidity and weak wind outweigh the sunny, hot outlook.
Log-probabilities: avoiding underflow
The lecture doesn't mention this, but every real implementation needs it. A naive Bayes score is a product of many numbers below 1. With 4 features that's harmless. A spam filter with 200 words, each with probability around 0.01, multiplies to . The smallest positive double-precision number is about ( with denormals), so the product underflows to exactly 0.0 for every class and the comparison becomes meaningless. (Verified: in Python, 0.01**200 returns 0.0.)
The fix: compare sums of logs instead of products. Since log is increasing, the argmax doesn't change:
For the 200-word email, the log score is about , a perfectly ordinary number.
For two classes, it's even nicer to subtract the two log scores. Each feature then contributes a log-likelihood ratio : positive values are evidence for Yes, negative for No. Here is the play-tennis day from Section 7 (natural logs):
With log scores and , the posterior is
the same as before. With more than two classes, subtract the largest score from all of them before exponentiating (the "log-sum-exp trick") so nothing overflows or underflows.
The log-odds is a sum of per-feature terms plus a constant. For binary features, each term is linear in the feature, so naive Bayes produces a linear classifier in the log-odds, the same form as logistic regression. The two differ only in how the weights are chosen: naive Bayes by counting within each class, logistic regression by directly maximizing . This is the cleanest example of a generative/discriminative pair.
Zero counts and Laplace smoothing
No overcast day in the training data is a No day, so . Any new overcast day gets a No score of exactly 0, whatever its other features say: one zero wipes out the whole product. The model claims to be 100% certain, from five examples. In log form the score is . Shalev-Shwartz & Ben-David §24.1.3 make the same point: an MLE of 0 for something that can happen gives infinite log-loss, a form of overfitting. And in text, zeros are everywhere (slide 32: by Zipf's law, about half of all words occur only once).
Take (Overcast, Hot, High, Strong). Hot, high humidity and strong wind all favour No. Yet the No score is , so .
The fix: add "virtual" counts (slides 18 and 23). Pretend every value was seen a few extra times. Laplace (add-one) smoothing adds 1 to each count and to the total, where is the number of values the attribute can take:
The general version is the m-estimate (slide 22, Mitchell §6.9.1.1):
- number of training examples with and
- number of training examples with
- prior guess for the probability, usually (uniform over the attribute's values)
- "equivalent sample size": how many virtual examples the prior is worth. gives back plain counting; gives Laplace
Yes:
No:
Now . Still Yes, since overcast days really are strongly associated with playing, but no longer a 100% certainty that ignores three pieces of evidence. (For the slide-21 day, smoothing moves from 0.205 to 0.280; the prediction stays No. scikit-learn's CategoricalNB(alpha=1) gives the same 0.280.)
Slides 18 and 23 write for any feature. The "+2" is only right for binary features. For an attribute with values, add to the denominator, or the smoothed probabilities won't sum to 1. Outlook on No days with "+2": . The spam slide (32) uses correctly, because each word there is present/absent (two values).
Slide 18 and the lecture (36:16, 44:29) say adding virtual counts is MAP estimation. That's true for a specific prior: with a Beta(2, 2) prior (Dirichlet with all parameters 2, for values), the MAP estimate is exactly . But with a uniform prior, MAP equals MLE and gives no smoothing at all. Shalev-Shwartz & Ben-David §24.5 derive the same formula a different way: as the full Bayesian prediction (averaging over all ) under a uniform prior, which they describe as adding "pseudoexamples". Either way, the idea is the same: the prior keeps estimates away from 0 and 1.
Quick check: Humidity has 2 values and was High on 4 of 5 No days. What's the Laplace estimate of P(Normal | No)?
, up from the raw . Smoothing pulls every estimate toward the uniform ; with only five examples, the pull is noticeable.
Continuous features: Gaussian naive Bayes
Counting doesn't work for a feature like temperature: almost every value is unique, so every count would be 0 or 1 (slide 24: "numberless values"). The common practice (slide 25) is to assume each continuous feature follows a normal (Gaussian) distribution within each class, and estimate its mean and standard deviation from that class's examples:
- mean of feature over the training examples of class
- standard deviation of feature over the training examples of class
Learning outputs Gaussians (one per feature per class) plus the class priors. Testing plugs the new values into each Gaussian and applies the MAP rule exactly as before (slide 24). Discrete and continuous features mix freely: each contributes its own factor.
Slide 29 writes . That's the variance ; the standard deviation needs a square root: . Its density is missing the 2 in the exponent; the correct exponent is (as on slide 24). The numbers on slides 28 and 30 were computed correctly, so use the correct formulas, not slide 29's.
The slides divide by (the "sample" standard deviation). The maximum-likelihood estimate in Shalev-Shwartz & Ben-David §24.1.1 divides by , and they note it is slightly biased. scikit-learn's GaussianNB also uses . For Yes-day temperatures the two give 6.16 vs 5.81. Either is acceptable; just be consistent. In the example below, using changes from 0.208 to 0.194, and the prediction stays No.
Temperature and humidity are now numbers. From the 9 Yes and 5 No days:
| Temp. mean | Temp. sd | Humidity mean | Humidity sd | |
|---|---|---|---|---|
| Yes | 73.0 | 6.2 | 79.1 | 10.2 |
| No | 74.6 | 7.9 | 86.2 | 9.7 |
Classify (Sunny, temperature 66, humidity 90, windy = true).
- Temperature density for Yes: For No (): 0.0279.
- Humidity density for Yes: 0.0221. For No: 0.0380.
- Yes score:
- No score:
- Predict No, with
Densities, not probabilities: a Gaussian density can exceed 1 when σ is small. That's fine, because the same kind of factor appears for every class and only the comparison matters.
Slide 30 lists the No-day temperature density as 0.0291. It should be 0.0279 (, , ). The slide's final answer, 0.000136, is correct and was evidently computed with 0.0279; with 0.0291 you'd get 0.000142. Also, the slide calls these products "likelihood of Yes/No", but they include the prior, so they're likelihood × prior.
The 2-D version (slides 26–27, 46:26–50:06): tell adults from children using height and weight, with 4 adults and 12 children. Priors are and . Fit four Gaussians (height and weight for each class), assume height and weight independent within each class, and compute
Because of the independence assumption, each class's contours are axis-aligned ellipses, not tilted ones, which is what slide 27's drawing shows.
The lecture suggests the Gaussian assumption becomes valid "when you have a large number of data ... based on the large number theory." That's a misreading of the central limit theorem, which says averages of many independent values become Gaussian. The estimated mean does become nearly Gaussian, but the feature itself keeps whatever shape it has. Income, word counts and waiting times stay skewed no matter how many you collect. If a feature is clearly non-Gaussian, transform it (e.g. take logs) or discretize it into bins.
Quick check: why does the Gaussian NB boundary come out as two thresholds rather than one?
Taking logs, each class's score is a quadratic in : . When the two classes have different , the terms don't cancel, so the difference is a quadratic with (up to) two roots. With equal variances, the terms cancel and there is a single threshold.
Application: spam filtering
Naive Bayes' most famous use. Each attribute is a word in the vocabulary, and its value is 1 if the word appears in the email and 0 if not. With 4 spam and 2 ham ("ham" = legitimate email) training messages:
| Text | Class | |
|---|---|---|
| D1 | send us your password | spam |
| D2 | send us your review | ham |
| D3 | review your password | ham |
| D4 | review us | spam |
| D5 | send your password | spam |
| D6 | send us your account | spam |
Priors: , . Each word's probability is the fraction of that class's emails containing it:
| Word | P(word | spam) | P(word | ham) | Laplace, spam | Laplace, ham |
|---|---|---|---|---|
| password | 2/4 | 1/2 | 3/6 | 2/4 |
| review | 1/4 | 2/2 | 2/6 | 3/4 |
| send | 3/4 | 1/2 | 4/6 | 2/4 |
| us | 3/4 | 1/2 | 4/6 | 2/4 |
| your | 3/4 | 2/2 | 4/6 | 3/4 |
| account | 1/4 | 0/2 | 2/6 | 1/4 |
To score the new email "review us now", every vocabulary word contributes: if present, if absent ("now" isn't in the vocabulary, so it's ignored). This present/absent model is called Bernoulli naive Bayes.
The ham likelihood is 0 because every ham email contained "your" and this one doesn't. Unsmoothed, the answer is spam with certainty. With Laplace smoothing (, +2 in the denominator since each word is binary), the scores become for spam and for ham, giving .
Slide 31 lists P(your | ham) = 1/2, but both ham emails (D2 "send us your review", D3 "review your password") contain "your", so it's 2/2. With the slide's 1/2, and (the slide rounds to 0.87). The slide's point survives: "note identical example." The new email is identical to D4, a spam training email, yet naive Bayes leans toward ham (0.88 on the slide, 0.52 with correct counts and smoothing). Independent word-by-word evidence doesn't memorize examples.
Weaknesses (slide 32): zero frequencies are very common in text (fix with smoothing, or replace with the word's overall frequency), and word independence is clearly false (phrases). The professor considers it an acceptable simplification (52:27). Spammers exploit it by stuffing spam with "hammy" words, each of which adds evidence for ham.
Mitchell §6.10 uses a different naive Bayes for text: one attribute per word position, with all positions sharing the same word distribution (the "multinomial" model), estimated as . It counts how many times each word occurs and ignores absent words. The slides' version (from Lavrenko) uses one binary attribute per vocabulary word. Both are naive Bayes; they just define the features differently. scikit-learn calls them MultinomialNB and BernoulliNB.
Missing values: just leave the feature out
Sometimes a feature is unknown: an applicant's credit history, or a medical test that wasn't done. Naive Bayes handles this with no special machinery: drop that feature's factor and compute the likelihood from the observed features only. Why this is correct (slide 34): if is unknown, the right thing is to sum over all its possible values. Under the naive assumption, only one factor depends on , and it sums to 1:
Slide 34's coin example: three tosses with the middle one unknown, (H, ?, T). The event is {HHT} or {HTT}, so its probability is
Classify (Sunny, ?, High, Strong). Skip the temperature factor:
Yes: . No: .
Predict No, with . (With the temperature known to be Cool it was 0.205; cool weather was mild evidence for Yes, and now it's gone.)
The professor contrasts this with k-NN (54:08): a distance can't be computed with a missing coordinate, so k-NN must first guess (impute) the value. Naive Bayes never has to.
The outline on slide 2 says "Missing values: fill the average". What the lecture and slides 33–34 actually teach is the opposite: don't fill anything in, just ignore the attribute ("no need to 'fill in' or explicitly model missing values"). Filling in the average is what you'd do for k-NN, and it isn't even defined for a categorical feature like Outlook.
When to use naive Bayes
- Training is one pass of counting: each attribute in each class is handled separately. No optimization, no learning rate, no local minima.
- Prediction is table lookups (or Gaussian evaluations) and a sum of logs, so it's very fast and scales to huge vocabularies.
- Competitive accuracy even when the independence assumption is violated, especially for text (spam filtering, document categorization). Mitchell §6.10 reports 89% accuracy classifying newsgroup posts into 20 groups.
- Interpretable: each feature's log-likelihood ratio says how much it pushed toward each class (the professor lists this as a benefit of generative models, 56:08).
- Handles missing values for free, and mixes discrete and continuous features.
- Weaknesses: its probabilities are often overconfident (correlated features are double-counted), it needs smoothing, and the Gaussian assumption can be badly wrong. It is a good base learner for ensembles (slide 35), which are coming later in the course.
In scikit-learn: CategoricalNB (discrete attributes, alpha = the smoothing count), GaussianNB (continuous), BernoulliNB (binary word presence), MultinomialNB (word counts).
Cheat sheet
| Bayes' rule | : posterior ∝ likelihood × prior |
| Evidence | ; same for every class |
| MAP decision | ; the ML rule drops |
| Generative / discriminative | model and / model directly |
| MLE / MAP estimate | / |
| Naive assumption | : features independent given the class |
| Classifier | |
| Parameters | binary features, 2 classes: vs for the full joint |
| Training (MLE) | fraction of records; = fraction of class- records with |
| Laplace smoothing | , = number of values of the attribute |
| m-estimate | , usually |
| Gaussian NB | , with per feature per class |
| Normalize | divide each class score by the sum of all class scores |
| Missing feature | drop its factor (it sums to 1) |
| Play tennis | (Sunny, Cool, High, Strong): 0.0053 vs 0.0206 → No, |
Practice
Q1Which pair of classifiers is one generative and one discriminative, in that order?
Q2A disease affects 1% of people. A test is positive for 90% of sick people and for 5% of healthy people. You test positive. What is P(sick | positive)?
Q3The naive Bayes assumption is that:
Q4A binary classification problem has 20 binary features. How many free parameters does naive Bayes estimate?
Q5In the unsmoothed play-tennis model, what is P(Yes | Overcast, Hot, High, Strong)?
Q6Outlook has 3 values. Overcast appears on 0 of the 5 No days. What is the Laplace-smoothed P(Overcast | No)?
Q7Why do implementations sum log-probabilities instead of multiplying probabilities?
Q8Estimating a binary feature's probability as (count + 1)/(n + 2) corresponds to:
Q9Using temperature alone, Yes days are N(73, 6.2²) with prior 9/14 and No days N(74.6, 7.9²) with prior 5/14. What is predicted for 95°F?
Q10A test example is missing its Humidity value. Naive Bayes should:
Problem 1: Classify (Rain, Hot, High, Weak) with the play-tennis data, first with plain counts and then with Laplace smoothing. Give the normalized P(Yes | x) each time.
Plain counts. From the slide-20 tables:
Yes: .
No: .
Predict No:
Laplace smoothing (+3 for Outlook and Temperature, +2 for Humidity and Wind):
Yes: .
No: .
Still No, but less confidently: . Smoothing pulls every estimate toward uniform, which pulls the posterior toward the prior split.
Checked in Python with exact fractions. Note the priors stay 9/14 and 5/14: only the conditional tables are smoothed.
Problem 2: Gaussian NB by hand. Children's heights are 120, 130, 140 cm; adults' are 165, 175, 185 cm. (a) Fit the Gaussians (use n − 1, as the slides do). (b) With equal priors, find P(child | h = 150) and the decision boundary. (c) Where does the boundary move if P(child) = 0.75?
(a) Children: , , so . Adults: , .
(b) Equal means the normalizing constants cancel. The log-likelihood ratio at 150 is
so . (Directly: densities 0.00540 and 0.00175.) The boundary is where the ratio is 0: , i.e. , the midpoint, as expected with equal variances and equal priors.
(c) Add to the log-ratio. Since , the boundary solves :
The boundary moves toward the adults: when children are more common, a borderline height is more likely a child. At , rises to about 0.902.
All values checked with scipy.stats.norm.