Generative vs discriminative models: Question bank

Before we start, one reminder of the symbols so nothing here is unearned:

Definition Symbol refresher (open if any notation feels unfamiliar)

$X$ = the features (pixels, word counts — the stuff we observe).
$y$ = the label / class (cat, dog, spam — the thing we want to know).
$P (y ∣ X)$ = "given the features, chance of each class" — read the bar | as "given".
$P (X ∣ y)$ = "given the class, chance of seeing these features".
$P (y)$ = the prior, the base rate of each class before you look at anything.
$\sum_{y^{'}}$ = "add this up over every possible class $y^{'}$ " — a total across all classes.
$P (X)$ = the evidence, the total chance of seeing features $X$ across all classes. Explicitly $P (X) = \sum_{y^{'}} P (X ∣ y^{'}) P (y^{'})$ : take each class's likelihood, weight it by that class's prior, and add them all up. It is exactly the denominator that turns a raw score into a probability.
$σ$ = the logistic sigmoid $σ (z) = \frac{1}{1 + e ^{- z}}$ : a smooth S-shaped squash that maps any real number $z$ to a probability in $(0, 1)$ — large positive $z \to 1$ , large negative $z \to 0$ , $z = 0 \to 0.5$ . Logistic regression feeds $z = w^{⊤} X + b$ into it to output $P (y = 1∣ X)$ .

One picture to hold in your head

The single most important visual for this whole topic is what each model draws. A generative model paints the two class-conditional bumps $P (X ∣ y)$ — the full shape of where each class's data lives — and the decision boundary falls out where the bumps cross. A discriminative model draws only that crossing line and ignores the bumps entirely.

Figure — Generative vs discriminative models

The second recurring picture is the asymptotic accuracy contrast: as labelled data grows, the discriminative curve climbs to a higher plateau because it spends every parameter on the boundary, while the generative model, weighed down by fitting whole distributions, plateaus lower.

True or false — justify

A generative model must always be more accurate than a discriminative one because it models more

False. Modelling the whole distribution

P (X ∣ y)

spends parameters on structure that classification never uses; with abundant labelled data a discriminative model that only fits

P (y ∣ X)

usually reaches higher asymptotic accuracy (the higher plateau in figure s02).

Naive Bayes assumes the features are independent of each other

False — it assumes they are independent given the class, i.e.

P (X ∣ y) = \prod_{j} P (x_{j} ∣ y)

. Across classes the features can be strongly correlated; the model just ignores correlation within a single class.

Logistic regression uses Bayes' theorem internally to make predictions

False. It parameterises

P (y ∣ X)

directly as

σ (w^{⊤} X + b)

(that

σ

is the sigmoid from the refresher) and never estimates

P (X ∣ y)

P (y)

, so it skips the Bayes bridge entirely — though the identity still holds abstractly.

A model that outputs

P (y ∣ X)

is discriminative regardless of how it was trained

True in spirit — the defining trait is what quantity it models. If the final object is the posterior boundary (the right panel of s01) and there is no explicit

P (X ∣ y)

, it behaves discriminatively.

Only generative models can be trained on unlabelled data

Broadly true for the classic split: generative models can learn

P (X)

from unlabelled examples — the setting explored in 4.6.01-Semi-Supervised-Learning, where unlabelled points reveal where the class-conditional hills of s01 sit — while pure discriminative models need labels

y

to fit

P (y ∣ X)

The softmax output of a neural net is a generative distribution over classes

False. Softmax gives

P (y ∣ X)

— a distribution over labels given the image, which is discriminative. A generative model would instead give a distribution over images given a label.

Bayes' theorem belongs only to generative models

False. It is a mathematical identity true for any joint distribution; generative models use its right-hand side explicitly, discriminative ones model the left-hand side directly and never expand it.

A GAN is a purely generative system

Half-true. The generator is generative (produces

X

), but the discriminator that trains it is discriminative (judges real vs fake) — so 4.5.03-Generative-Adversarial-Networks is a hybrid: it draws a boundary (right panel of s01) to push a generator toward the true data hills (left panel).

Spot the error

P (y = 1∣ X_{n e w}) = P (X_{n e w} ∣ y = 1) \cdot P (y = 1)

— done." What is missing?

The normaliser. You must divide by the evidence

P (X_{n e w}) = \sum_{y^{'}} P (X_{n e w} ∣ y^{'}) P (y^{'})

so the posteriors over all classes sum to 1; without it you have an unnormalised score, not a probability.

Someone writes

P (X) = \sum_{y^{'}} P (X ∣ y^{'})

(no prior in the sum). Fix it

Each term must be weighted by its prior:

P (X) = \sum_{y^{'}} P (X ∣ y^{'}) P (y^{'})

. Otherwise you overcount rare classes as if they were as common as frequent ones.

"For Gaussian Naive Bayes I model

P (X ∣ y)

as one 784-dimensional Gaussian with full covariance." Why is that not Naive Bayes?

The naive assumption forces a product of per-pixel 1-D Gaussians,

\prod_{j} N (μ_{j k}, σ_{j k}^{2})

. A full covariance Gaussian models pixel correlations, so it is a different (non-naive) generative model.

"Discriminative models can't handle a missing feature, so we just set it to zero." Why is zero wrong?

Zero is a real value, not "unknown", and biases the boundary. A generative model handles it honestly by marginalising:

P (y ∣ X_{o b s}) \propto P (y) \int P (X_{o b s}, x_{mi ss} ∣ y) d x_{mi ss}

"Cross-entropy loss

L = - \sum_{i} lo g P (y_{i} ∣ X_{i})

trains a generative model." Where's the slip?

It maximises the conditional likelihood

P (y ∣ X)

, which is the discriminative objective. A generative training loss would instead maximise the joint

P (X, y)

or the marginal

P (X)

"Prior

P (y)

doesn't matter if I have lots of data." Correct it

With infinite data the estimated prior is accurate, but the prior still enters every posterior. If class balance in deployment differs from training, an unadjusted prior systematically biases predictions.

Why questions

Why does a generative classifier still need

P (y)

and not just

P (X ∣ y)

Because the decision uses the posterior

P (y ∣ X) \propto P (X ∣ y) P (y)

; the likelihood alone ignores that some classes are rarer, so a rare class with a slightly better fit could be wrongly preferred.

Why is the discriminative model usually more sample-efficient?

It only has to learn where classes separate (one line in s01), not the full shape of each class's data cloud — fewer effective parameters means it converges to a good boundary with less data.

Why can a generative model detect anomalies but a discriminative one struggles?

The generative model knows

P (X)

, so a genuinely weird input scores a low likelihood under every hill in s01; the discriminative model only knows the relative boundary and will confidently assign a nearest class even to nonsense.

Why does Naive Bayes work "surprisingly well" despite a clearly false independence assumption?

Classification only needs the argmax of the posterior, not calibrated probabilities. Even when the independence assumption skews the magnitudes, the ranking of classes often stays correct.

Why do we take the negative log of the probability in the training objective?

Products of many small probabilities underflow and are hard to optimise; the log turns products into sums, and negating converts "maximise probability" into a standard "minimise loss".

Why does generative modelling enable semi-supervised learning while pure discrimination does not?

Unlabelled points still inform

P (X)

(where data lives), which shapes each

P (X ∣ y)

region; discriminative training has no place to put a point with no label, so it simply ignores it — the exact leverage exploited in 4.6.01-Semi-Supervised-Learning.

Why might a generative model with a wrong distributional assumption underperform even with lots of data?

Model misspecification: if the true

P (X ∣ y)

isn't Gaussian (or isn't factorised), the fit converges to the best wrong shape, and its induced boundary is biased in a way more data cannot cure.

Edge cases

What happens to Naive Bayes if a test word never appeared in the spam training set?

Its estimated

P (x_{j} ∣ y = 1) = 0

zeroes the entire product

\prod_{j} P (x_{j} ∣ y)

, killing that class. The fix is Laplace/additive smoothing so no count is ever exactly zero.

You have equal priors

P (y = 0) = P (y = 1) = 0.5

. Does the prior still affect the decision?

No — equal priors cancel in the posterior ratio, so the decision reduces to comparing likelihoods

P (X ∣ y)

alone. This is the one case where "likelihood only" is legitimate.

The evidence

P (X) = 0

for some feature vector. What does that mean and can we classify it?

It means the model assigns that exact

X

zero total probability, so

P (y ∣ X)

0/0

— undefined. In practice this signals an out-of-distribution input; smoothing or a continuous density avoids the hard zero.

A class has zero training examples, so

P (y = k) = 0

. What does the model do with it?

That class's posterior is forced to zero everywhere (

P (X ∣ y = k) P (y = k) = 0

), making it unrecoverable at test time regardless of how well the features match — a reason to smooth priors too.

All features are identical across every class. What can a discriminative model learn?

Nothing useful — with no feature separation the best posterior is just the prior, so it collapses to always predicting the majority class. A generative model degrades to the same base-rate decision.

One feature perfectly predicts the label (separable data) in logistic regression. What breaks?

The weights

w

diverge toward infinity trying to push

σ (\cdot)

to exactly 0/1, so training never converges without regularisation — a discriminative-specific failure that generative models don't share. See 3.3.01-Logistic-Regression.

As training data

\to \infty

, which model wins and why?

Asymptotically the discriminative model wins on classification accuracy (the higher plateau in s02, because it targets the boundary directly), while the generative model may plateau lower because its distributional assumptions bias the fit — the classic efficiency-vs-asymptote tradeoff.