Worked examples — Generative vs discriminative models

This page is the hands-on drill room for the parent topic. We will not re-explain the theory from scratch — but we will re-earn every symbol the moment we use it, so a reader who lands here cold can still follow.

Before any worked example, here is the one-line meaning of the two objects everything spins around:

---

The scenario matrix

Every generative-vs-discriminative problem you'll be asked lives in one of these cells. The rest of the page hits every cell.

#	Cell (the case class)	What makes it tricky	Covered by
A	Two-class, one feature — the baseline	Getting the Bayes plumbing right	Ex 1
B	Sign / boundary case: $P(X\mid y=1)=P(X\mid y=0)$	Posterior collapses to the prior	Ex 2
C	Degenerate input: a word/feature never seen in a class ($P=0$)	Zero kills the whole product — need smoothing	Ex 3
D	Missing feature at test time	Generative marginalises; discriminative stalls	Ex 4
E	Continuous feature (Gaussian class model)	Density, not probability; compare ratios	Ex 5
F	Limiting behaviour: prior $\to 0$ or $\to 1$	One class swamps the posterior	Ex 6
G	Discriminative side: logistic regression from the same data	Same answer, different route	Ex 7
H	Real-world word problem (medical test)	Base-rate fallacy	Ex 8
I	Exam twist: recover $P(y\mid X)$ knowing only $P(X\mid y)$ ratios	Priors cancel or don't	Ex 9

Figure 0 below is the map of the machine every generative example on this page runs. Read it as a flow from the two inputs on the left (features $X$, teal; label $y$, plum) into the orange score box, then down into normalise. It has no axes because it is a data-flow diagram, not a plot; the colour of each box tells you whether it comes from the likelihood side (teal) or the prior side (plum). Every worked example below labels its steps "score" and "normalise" to match this figure.

The takeaway: every generative classifier is the same two-step machine — (1) score each class with $P(X\mid y)P(y)$, (2) normalise so the scores sum to 1. We will name those two steps explicitly in Ex 1 and then just point back to them.

---

Cell A — the baseline: two classes, one feature

1. Score each class (the "score" box of Figure 0) with $\text{score}(y)=P(X\mid y)P(y)$. Why this step? This is the numerator of Bayes' theorem, $P(y\mid X)=\frac{P(X\mid y)P(y)}{P(X)}$. The score is "likelihood $\times$ prior" — how much evidence votes for this class. $\text{score}(y=1)=0.8\times 0.4=0.32,\text{score}(y=0)=0.1\times 0.6=0.06$

2. Add the scores to get the evidence $P(X)$ (the "normalise" box, part 1). Why this step? $P(X)={\sum }_{y^{\prime }}P(X\mid y^{\prime })P(y^{\prime })$ — the email had to come from some class, so we sum over all of them. This is the denominator. $P(X=1)=0.32+0.06=0.38$

3. Divide to get the posterior (the "normalise" box, part 2). Why this step? Dividing by the total forces the two class-probabilities to add to $1$ — that's what makes them honest probabilities. $P(y=1\mid X=1)=\frac{0.32}{0.38}=0.8421\dots$

Verify: The other class must fill the rest: $P(y=0\mid X=1)=0.06/0.38\approx 0.158$, and $0.842+0.158=1.000$ ✓. Sanity: "free" is 8× more likely in spam, so pushing well above the $0.4$ prior is right.

---

Cell B — the boundary: when the feature says nothing

1. Score each class. Why this step? Same score box as Ex 1, but notice the two likelihoods are now identical ($0.5$ and $0.5$) — watch what that identical factor does when we divide. $\text{score}(y=1)=0.5\times 0.4=0.2,\text{score}(y=0)=0.5\times 0.6=0.3$

2. Add the scores to get the evidence $P(X)$. Why this step? Denominator $P(X)={\sum }_{y^{\prime }}P(X\mid y^{\prime })P(y^{\prime })$. Here every term carries the same likelihood $0.5$, so $0.5$ is a common factor of numerator and denominator — a strong hint it will cancel. $P(X=1)=0.2+0.3=0.5$

3. Divide to get the posterior. Why this step? The common likelihood $0.5$ cancels top and bottom, leaving only the prior structure. $P(y=1\mid X=1)=\frac{0.2}{0.5}=0.4$

Verify: $0.4$ equals the prior $P(y=1)=0.4$ exactly ✓. Algebra confirms it: if $P(X\mid y=1)=P(X\mid y=0)=c$, then $\frac{cP(y=1)}{cP(y=1)+cP(y=0)}=\frac{P(y=1)}{P(y=1)+P(y=0)}=P(y=1)$.

---

Cell C — degenerate input: the zero-probability trap

1. Write the naive-Bayes likelihood (features independent given class): $P(X\mid y)={\prod }_{j}P(x_j\mid y).$ Why this step? Naive Bayes multiplies each word's likelihood; that's how it combines evidence from many features.

2. Plug in the zero. $P(X\mid y=1)=\underset{0.8}{\underbrace{P(\text{free}\mid y=1)}}\times \underset{0}{\underbrace{P(\text{invoice}\mid y=1)}}=0$ Why this matters: One zero factor annihilates the entire product. So $\text{score}(y=1)=0\times P(y=1)=0$, and $P(y=1\mid X)=0$ — the model is infinitely certain it's not spam, on the strength of a single unseen word. That's a bug, not a feature.

3. See where the counts come from. Why this step? Before smoothing, each word estimate is literally a fraction of counts collected from data: $P(\text{word }j\mid y=1)=\frac{{\text{count}}_j}{N_1},N_1={\sum }_{j=1}^V{\text{count}}_j.$ Concretely, imagine the spam emails contained only $V=5$ distinct words with counts $[\text{free}:40,\text{deal}:30,\text{now}:15,\text{buy}:10,\text{invoice}:0]$. Then $N_1=40+30+15+10+0=95$ total word-slots, and "invoice" has count $0$ — that is exactly how the fatal $0$ arose.

4. Fix with Laplace (add-one) smoothing. Why this step? We pretend every one of the $V$ vocabulary words was seen one extra time, so no estimate is ever exactly $0$. Add $1$ to the numerator (the pretend extra sighting of this word) and add $V$ to the denominator (one pretend sighting for each of the $V$ words, so the smoothed estimates still sum to $1$ across the vocabulary): $P(\text{invoice}\mid y=1)=\frac{\text{count}+1}{N_1+V}=\frac{0+1}{95+5}=\frac{1}{100}=\mathrm{0.01.}$ Now $P(X\mid y=1)=0.8\times 0.01=0.008\ne 0$, and the posterior is finite and usable. In general you apply the same $\frac{{\text{count}}_j+1}{N_1+V}$ formula to every word $j$ and every class, not just the missing one — that keeps the whole set of estimates consistent.

Verify: Before smoothing $P(X\mid y=1)=0$ (product with a zero) ✓; the counts give $N_1=95$; smoothed word estimate $\frac{0+1}{95+5}=0.01$ and after smoothing $0.8\times 0.01=0.008>0$ ✓.

---

Cell D — missing feature: generative marginalises, discriminative stalls

1. Marginalise out the missing feature. Why this step? A generative model knows the joint $P(x_1,x_2\mid y)$, so to "forget" the unreadable $x_2$ we sum over every value it could have taken: ${\sum }_{x_2}P(x_1,x_2\mid y)=P(x_1\mid y).$ Under naive independence $P(x_1,x_2\mid y)=P(x_1\mid y)P(x_2\mid y)$, and ${\sum }_{x_2}P(x_2\mid y)=1$ (probabilities of all outcomes of $x_2$ add to one), so the $x_2$ factor cleanly disappears, leaving just $P(x_1\mid y)$.

2. Why a discriminative model can't do this (the derivation). Why this matters: A discriminative model stores only the function $P(y\mid x_1,x_2)$. To drop $x_2$ honestly it would need $P(y\mid x_1)={\sum }_{x_2}P(y\mid x_1,x_2)P(x_2\mid x_1),$ which is a weighted average requiring the factor $P(x_2\mid x_1)$ — a piece of the input distribution the discriminative model never learned (it only ever modelled $y$ given the inputs, never how the inputs are distributed). With $P(x_2\mid x_1)$ unavailable, it cannot form this sum; plugging a guessed or zero value for $x_2$ is unprincipled. The generative model, storing $P(x_2\mid y)$, has the missing ingredient — that is the whole advantage.

3. Score each class with the marginalised likelihood. Why this step? We are back to the ordinary score box, now feeding it $P(x_1\mid y)$ from step 1 instead of a full joint. $\text{score}(y=1)=0.8\times 0.5=0.4,\text{score}(y=0)=0.2\times 0.5=0.1$

4. Add and divide (normalise). Why this step? Denominator $P(x_1=1)=0.4+0.1=0.5$ forces the posteriors to sum to $1$. $P(y=1\mid x_1=1)=\frac{0.4}{0.5}=0.8$

Verify: $P(y=0\mid x_1=1)=0.1/0.5=0.2$; sum $=1$ ✓. The generative model produced a clean answer using marginalisation — exactly the "handles missing features" row of the parent's comparison table.

---

Cell E — continuous feature: Gaussian class-conditionals

Figure 1 below plots the two class densities and the test point. Reading it: the horizontal axis is the diameter $x$ in cm; the vertical axis is the density $p(x\mid y)$ (a height, not a probability, per the definition above). The orange curve is apples (${\mu }_A=7$), the teal curve is oranges (${\mu }_O=9$); the two dotted vertical lines mark their means and the plum dashed line marks the test point $x=8$, where both curves reach the same height.

1. Use the Gaussian density for each class-conditional. Why a density, not a probability? Because $x$ is continuous (see the definition box), so we compare densities $p(x\mid y)=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp (-\frac{(x-\mu {)}^2}{2{\sigma }^2})$. Both $\sigma$ are equal, so the constant $\frac{1}{\sqrt{2\pi }}$ will cancel in the ratio, leaving only the exponentials — this is why comparing densities is legitimate even though each single point's probability is zero.

2. Evaluate the exponents at $x=8$. Why this step? With equal $\sigma$ the comparison reduces to the squared distance $(x-\mu {)}^2$ inside each exponential — whichever mean is closer wins. On Figure 1 the plum dashed line at $x=8$ hits both curves at the same height: $(x-{\mu }_A{)}^2=(8-7{)}^2=1,(x-{\mu }_O{)}^2=(8-9{)}^2=1.$ Both exponents equal $-\frac{1}{2}$, so $p(x\mid A)=p(x\mid O)$.

3. Normalise with equal priors. Why this step? The posterior is $\text{score}/(\text{sum of scores})$ as always; with equal priors the priors cancel too, leaving only the (equal) densities. $P(y=A\mid x=8)=\frac{0.5p(x\mid A)}{0.5p(x\mid A)+0.5p(x\mid O)}=\frac{1}{1+1}=0.5$

Verify: Both densities equal $\frac{1}{\sqrt{2\pi }}{e}^{-1/2}$; their ratio is $1$; posterior $=0.5$ ✓. Move to $x=7.9$ and apples must win (closer to ${\mu }_A$).

---

Cell F — limiting behaviour: a lopsided prior

1. Score each class with the tiny prior. Why this step? Only the prior $P(y)$ changed from Ex 1 — same likelihoods $0.8$ and $0.1$ — so recomputing the scores isolates the effect of the prior alone. $\text{score}(y=1)=0.8\times 0.01=0.008,\text{score}(y=0)=0.1\times 0.99=0.099$

2. Add and divide (normalise). Why this step? Denominator $P(X=1)=0.008+0.099=0.107$; dividing gives an honest posterior. $P(y=1\mid X=1)=\frac{0.008}{0.107}\approx 0.0748$

Verify: $0.099/0.107\approx 0.925$ for ham; $0.0748+0.925\approx 1.000$ ✓. And it is far below the $0.84$ of Ex 1 — the only thing we changed was the prior, and it moved the answer the expected direction ✓.

---

Cell G — the discriminative route to the same answer

1. Compute the target logit from the answer we already have. Why this step? $\sigma$ is invertible: the log-odds $z=\ln \frac{P(y=1\mid X)}{P(y=0\mid X)}$. Discriminative and generative models must agree on this if they agree on the posterior. $z=\ln \frac{0.842}{0.158}=\ln (5.333\dots )=1.6740\dots$

2. Check it against the generative log-odds. Why this step? Bayes gives $z=\ln \frac{P(X\mid y=1)P(y=1)}{P(X\mid y=0)P(y=0)}$ (the $P(X)$ cancels inside the ratio). Plug Ex-1 numbers: $z=\ln \frac{0.8\times 0.4}{0.1\times 0.6}=\ln \frac{0.32}{0.06}=\ln (5.333\dots )=1.6740\dots$

3. Squash back. Why this step? Applying $\sigma$ undoes the log-odds and returns to a probability, closing the loop. $\sigma (1.6740)=\frac{1}{1+e^{-1.6740}}=0.842$

Verify: Both routes give $z=\ln (16/3)=1.6740\dots$ and $\sigma (z)=0.842$, matching Ex 1 ✓.

---

Cell H — real-world word problem: the base-rate trap

1. Set up as generative classification — this is Bayes' theorem. Why this framing? $D$/$\neg D$ are the labels $y$; the test result $+$ is the feature $X$. We want the posterior $P(D\mid +)$.

2. Score both classes with $\text{score}(y)=P(+\mid y)P(y)$. Why this step? Feed the score box the two priors and two likelihoods. The healthy group is enormous, so even a small $5\%$ false-positive rate produces more positives than the tiny sick group does — watch the numbers. $\text{score}(D)=P(+\mid D)P(D)=0.99\times 0.001=0.00099$ $\text{score}(\neg D)=P(+\mid \neg D)P(\neg D)=0.05\times 0.999=0.04995$

3. Add and divide (normalise). Why this step? Denominator $P(+)=0.00099+0.04995=0.05094$ is the total probability of a positive test; dividing gives the honest posterior. $P(D\mid +)=\frac{0.00099}{0.05094}\approx 0.01943$

Verify: $P(D\mid +)=0.00099/0.05094\approx 0.0194$; the complement $P(\neg D\mid +)\approx 0.9806$; sum $=1$ ✓.

---

Cell I — exam twist: only likelihood ratios given

1. Write the posterior as odds. Why this step? When only a ratio is known, work in odds — the unknown $P(X)$ cancels: $\frac{P(y=1\mid X)}{P(y=0\mid X)}=\underset{\text{likelihood ratio}=3}{\underbrace{\frac{P(X\mid y=1)}{P(X\mid y=0)}}}\times \underset{\text{prior odds}}{\underbrace{\frac{P(y=1)}{P(y=0)}}}.$

2. Equal priors → prior odds $=1$: Why this step? With $P(y=1)=P(y=0)=0.5$ their ratio is $1$, so posterior odds $=$ likelihood ratio. $\text{posterior odds}=3\times 1=3\Rightarrow P(y=1\mid X)=\frac{3}{3+1}=\mathrm{0.75.}$

3. Now shift the prior to $P(y=1)=0.25$ → prior odds $=\frac{0.25}{0.75}=\frac{1}{3}$: Why this step? The stronger evidence ($3\times$) and the weaker prior ($\frac{1}{3}$) exactly cancel — the twist the examiner is testing. $\text{posterior odds}=3\times \frac{1}{3}=1\Rightarrow P(y=1\mid X)=\frac{1}{1+1}=\mathrm{0.5.}$

Verify: Equal-prior case $3/4=0.75$ ✓. Skewed-prior case posterior odds $=3\cdot \frac{1}{3}=1\Rightarrow 0.5$ ✓.

---

Wrap-up

Recall Did we cover every cell?

A baseline (Ex 1), boundary/uninformative feature (Ex 2), zero-probability degenerate (Ex 3), missing feature (Ex 4), continuous Gaussian (Ex 5), prior limit (Ex 6), discriminative equivalent (Ex 7), real-world base-rate (Ex 8), exam ratio twist (Ex 9). Every row of the scenario matrix is hit. ✓

Recall Quick self-test

A feature is equally likely in both classes. What does the posterior equal? ::: The prior — the feature carries no information (Cell B). A word appears zero times in a class. What breaks and what fixes it? ::: The likelihood product becomes $0$; Laplace/add-one smoothing fixes it (Cell C). How does a generative model handle a missing feature? ::: By marginalising (summing/integrating) it out of $P(X\mid y)$ (Cell D). Why can't a discriminative model marginalise a missing input? ::: It never learned the input distribution $P(x_2\mid x_1)$ needed to average over the missing value (Cell D). Why do we write $p(x\mid y)$ not $P(x\mid y)$ for a diameter? ::: Continuous values have zero point-probability, so we use a density (a height whose area gives probability) (Cell E). "99% accurate test, positive result" — biggest thing people forget? ::: The base rate / prior $P(D)$ (Cell H).

Connections

• Generative vs Discriminative Models — parent topic these examples drill
• 4.1.01-Bayes-Theorem — the plumbing behind every score-and-normalise
• 4.5.04-Naive-Bayes-Classifier — the generative machine used in Ex 1–4
• 3.3.01-Logistic-Regression — the discriminative route in Ex 7
• 4.5.05-Gaussian-Mixture-Models — continuous class-conditionals as in Ex 5
• 4.5.02-Variational-Autoencoders, 4.5.03-Generative-Adversarial-Networks — deep generative successors
• 4.6.01-Semi-Supervised-Learning — where marginalisation (Ex 4) pays off