Exercises — Generative vs discriminative models

(a) D — models $P(y|X)$ straight through the sigmoid. See 3.3.01-Logistic-Regression. (b) G — models $P(X|y)$ and $P(y)$, then applies Bayes. See 4.5.04-Naive-Bayes-Classifier. (c) D — softmax outputs $P(y|X)$ directly. (d) G — models the density $P(X)$ (as a mixture of bumps). See 4.5.05-Gaussian-Mixture-Models. (e) D — learns a boundary (a hyperplane maximizing margin), no data density.

(a) The posterior $P(y|X)$ directly. (b) The likelihood $P(X|y)$ and the prior $P(y)$. From these it reconstructs the posterior via Bayes' theorem.

WHAT we do: apply Bayes with a two-term denominator (marginalize over spam and ham). WHY: the email could have come from either class; we sum both possibilities for the evidence $P(\text{free})$. $P(\text{spam}\mid \text{free})=\frac{P(\text{free}\mid \text{spam})P(\text{spam})}{P(\text{free}\mid \text{spam})P(\text{spam})+P(\text{free}\mid \text{ham})P(\text{ham})}$ Numerator: $0.8\times 0.4=0.32$. Denominator: $0.32+(0.1\times 0.6)=0.32+0.06=0.38$. $P(\text{spam}\mid \text{free})=\frac{0.32}{0.38}\approx \mathrm{0.842.}$ So one word "free" pushes belief from the prior $0.4$ up to about $0.842$.

WHAT: multiply per-word likelihoods (Naive Bayes assumes features independent given the class). WHY the product: $P(X\mid y)={\prod }_{j}P(x_j\mid y)$ — this is the "naive" step. It is independence conditional on $y$, not overall. Spam score: $P(\text{free}\mid s)P(\text{meeting}\mid s)P(s)=0.8\times 0.05\times 0.4=0.016$. Ham score: $0.1\times 0.5\times 0.6=0.03$. $P(\text{spam}\mid X)=\frac{0.016}{0.016+0.03}=\frac{0.016}{0.046}\approx \mathrm{0.348.}$ The word "meeting" (strongly ham) overturned the earlier lean toward spam — belief drops to about $0.348$.

With only 50 examples, discriminative models are usually safer for pure classification. WHY: a full generative model must estimate the entire distribution $P(X|y)$ for each class — for high-dimensional $X$ that can be a huge number of parameters (means, variances, and off-diagonal covariances). With 50 points you cannot estimate all of them reliably; the estimates are noisy and overfit. Important nuance: not every generative model is this heavy. Naive Bayes makes the conditional-independence assumption $P(X|y)={\prod }_{j}P(x_j|y)$, which throws away all off-diagonal covariances and keeps only a mean (and maybe a variance) per feature per class. That drastically cuts the parameter count — often to the same order as a discriminative model. So the "generative needs more data" rule is really about how much structure the generative model tries to capture: a full-covariance Gaussian is data-hungry; Naive Bayes can actually be very data-efficient (sometimes beating logistic regression at tiny sample sizes). A discriminative model only needs to place the boundary, which is a small object (a hyperplane needs $d+1$ numbers, not a full density). Fewer parameters = it can be pinned down with less data. Caveat that shows real understanding: if many unlabeled images are also available, a generative model can exploit them (it can score $P(X)$ on unlabeled data), flipping the advantage — see 4.6.01-Semi-Supervised-Learning.

A generative model owns $P(X|y)=P(x_1,x_2,x_3,\dots |y)$. If $x_3$ is missing, we integrate it out: $P(x_1,x_2,x_4,\cdots \mid y)=\int P(x_1,x_2,x_3,x_4,\cdots \mid y)d{x}_3.$ WHAT this is: summing the joint density over every possible value the missing feature could have taken. WHY it works: because the generative model knows the joint distribution, "not knowing $x_3$" is expressible as "average over $x_3$". For Naive Bayes it's even simpler — the missing factor just drops out of the product ${\prod }_{j}P(x_j|y)$. Logistic regression only owns the conditional $P(y|X)$, a single function that requires all inputs $x_1,\dots ,x_n$ to be present to compute $w^{\top }X$. It has no distribution over $x_3$ to average against, so a missing feature breaks the dot product — you must impute a value first, which is an outside patch, not a native capability.

Start from Bayes for the binary case: $P(y=1|x)=\frac{P(x|1)P(1)}{P(x|1)P(1)+P(x|0)P(0)}=\frac{1}{1+\frac{P(x|0)P(0)}{P(x|1)P(1)}}.$ WHY this rewrite: dividing top and bottom by the $y=1$ term exposes a single ratio — that ratio is what will become the sigmoid's exponent. Now let $P(x|y)=\frac{1}{\sqrt{2\pi }\sigma }\exp (-\frac{(x-{\mu }_y{)}^2}{2{\sigma }^2})$. WHY the normalizing constants cancel: the prefactor $\frac{1}{\sqrt{2\pi }\sigma }$ is identical for both classes because they share the same $\sigma$. So when we form the ratio $\frac{P(x|0)}{P(x|1)}$, the two prefactors are the same number top and bottom and divide out exactly — leaving only the ratio of the exponentials. (If $\sigma$ differed between classes, the prefactors would not match and would survive as an extra constant — this is exactly what breaks the clean logistic form; see the edge-case note below.) With the prefactors gone, $\frac{P(x|0)}{P(x|1)}=\exp (-\frac{(x-{\mu }_0{)}^2-(x-{\mu }_1{)}^2}{2{\sigma }^2}).$ We want the whole posterior in the form $\sigma (wx+b)=\frac{1}{1+e^{-(wx+b)}}$, so the denominator's exponent must equal $-(wx+b)$. The denominator is $1+\frac{P(x|0)P(0)}{P(x|1)P(1)}=1+\exp (-\frac{(x-{\mu }_0{)}^2-(x-{\mu }_1{)}^2}{2{\sigma }^2})\cdot \frac{P(0)}{P(1)}$. Write the prior ratio as $\frac{P(0)}{P(1)}=\exp \left(\ln \frac{P(0)}{P(1)}\right)$ so the whole denominator becomes $1+\exp (-(wx+b))$ where $-(wx+b)=-\frac{(x-{\mu }_0{)}^2-(x-{\mu }_1{)}^2}{2{\sigma }^2}+\ln \frac{P(0)}{P(1)}.$ Expand: $(x-{\mu }_0{)}^2-(x-{\mu }_1{)}^2=-2x({\mu }_0-{\mu }_1)+({\mu }_0^2-{\mu }_1^2)$, which is linear in $x$. Dividing by $-2{\sigma }^2$ and matching coefficients of $x$ and the constant to $-(wx+b)=-wx-b$: $w=\frac{{\mu }_1-{\mu }_0}{{\sigma }^2},b=\frac{{\mu }_0^2-{\mu }_1^2}{2{\sigma }^2}+\ln \frac{P(1)}{P(0)}.$ (The prior term is $+\ln \frac{P(1)}{P(0)}=-\ln \frac{P(0)}{P(1)}$ — note the sign: the $\ln \frac{P(0)}{P(1)}$ that appeared in $-(wx+b)$ flips when we solve for $b$ itself.) Substituting back: $P(y=1|x)=\frac{1}{1+e^{-(wx+b)}}=\sigma (wx+b).$ The punchline: a generative Gaussian model, under the shared-variance assumption, yields exactly the discriminative logistic form. They are two doors into the same room — but the generative one made an extra assumption (Gaussian, shared $\sigma$) to get there, while logistic regression fits $w,b$ freely. See 3.3.01-Logistic-Regression.

Edge case — unequal variances: this whole result hinges entirely on the shared-variance assumption. If ${\sigma }_0\ne {\sigma }_1$, two things break: (1) the prefactors no longer cancel, and (2) the $x^2$ terms in $(x-{\mu }_0{)}^2/{\sigma }_0^2-(x-{\mu }_1{)}^2/{\sigma }_1^2$ no longer cancel either — leaving an $x^2$ term in the exponent. The posterior is then not a simple logistic $\sigma (wx+b)$ but a sigmoid of a quadratic $\sigma (a{x}^2+wx+b)$, giving a curved (possibly two-piece) boundary. Exercise 5.1 works exactly this unequal-variance case.

• Generator: generative — it learns to produce samples $X$, i.e. it models (implicitly) $P(X)$ by mapping noise $\to$ data.
• Discriminator: discriminative — it learns $P(\text{real}\mid X)$, a boundary between real and fake. The system is called generative because its deliverable is new data from the generator; the discriminator is merely a training critic. This is why the parent note lists GANs as a "hybrid": the learning signal comes from a discriminative component, but the goal is generation. Contrast with VAEs, which are generative through an explicit likelihood.

(a) The boundary is where the two weighted likelihoods are equal (priors are equal, so they cancel): $\frac{1}{\sqrt{2\pi \cdot 1}}{e}^{-x^2/2}=\frac{1}{\sqrt{2\pi \cdot 4}}{e}^{-x^2/8}.$ Take logs (monotone, safe): $-\frac{1}{2}\ln (2\pi )-\frac{x^2}{2}=-\frac{1}{2}\ln (8\pi )-\frac{x^2}{8}.$ Collect the $x^2$ terms on one side and constants on the other: $-\frac{x^2}{2}+\frac{x^2}{8}=\frac{1}{2}\ln (2\pi )-\frac{1}{2}\ln (8\pi )=\frac{1}{2}\ln \frac{2\pi }{8\pi }=\frac{1}{2}\ln \frac{1}{4}=-\ln 2.$ Left side $=-\frac{3x^2}{8}$. So $-\frac{3x^2}{8}=-\ln 2$. Solving this quadratic in $x$ gives $x^2=\frac{8\ln 2}{3}\approx 1.848$, hence two threshold points $x=+\sqrt{\frac{8\ln 2}{3}}\approx +1.359,x=-\sqrt{\frac{8\ln 2}{3}}\approx -\mathrm{1.359.}$ (b) The boundary condition is quadratic in $x$, and solving that quadratic yields the two threshold points above. These two thresholds carve the line into three pieces. The class-1 region is the union of the outer two pieces (a disconnected region). Why each piece belongs where it does: the wider Gaussian (class 1, variance 4) decays slowly, so it dominates in the tails; the narrower one (class 0) is tall near the center, so it dominates there. Writing $t=\sqrt{\frac{8\ln 2}{3}}\approx 1.359$, the explicit regions are:

• Class 0: $-t<x<t$, i.e. the open interval $(-1.359,1.359)$ — the center belongs to the tall, narrow bump.
• Class 1: $x\le -t$ or $x\ge t$, i.e. $(-\infty ,-1.359]\cup [1.359,\infty )$ — the two tails belong to the flat, wide bump.

So class 1's region is two half-lines — not a single interval. That disconnectedness is the whole point: unequal variances give curved boundaries with possibly two crossings, unlike the single-line boundary of Exercise 4.1.

The figure below traces exactly this: the two red dashed lines are the thresholds $\pm 1.359$, the blue-shaded center is class 0, and the two green-shaded tails are the disconnected class-1 region.

(a) G — only generative models can synthesize $X$ (VAE/GAN). (b) D — abundant labels + pure classification → the sample-efficient, boundary-focused choice wins asymptotically. (c) G — the huge unlabeled pool can be scored under $P(X)$; this is exactly 4.6.01-Semi-Supervised-Learning's home turf. (d) G — missing features are handled natively by marginalizing them out of $P(X|y)$ (Exercise 3.2).

Which side of the boundary owns the tails when class variances differ? ::: The class with the larger variance (its density decays slower, so it wins far from the mean). A generative Gaussian with shared variance reduces to which discriminative model? ::: Logistic regression (sigmoid posterior). Why can generative models use unlabeled data? ::: They model $P(X)$, so unlabeled points still contribute to fitting the density.