Generative vs discriminative models: Visual walkthrough

Step 1 — Two clouds of dots

WHAT. Imagine a sheet of paper. Every animal we saw becomes a single dot. Its horizontal position is one measurement (say "size"), its vertical position another (say "spottiness"). We colour a dot lavender if it was actually a leopard and coral if it was a housecat.

WHY. Before any maths, a "model" is just an attempt to make sense of this picture. Everything we write later is a description of where the dots sit. If you can see the two clouds, you already understand the problem.

PICTURE. Look below: two soft blobs of dots. They overlap a little in the middle — that overlap is the hard region, where even an expert hesitates.

Figure — Generative vs discriminative models

Step 2 — The question we actually want answered

WHAT. A brand-new grey dot lands on the paper. We want a number: how sure are we it's a leopard? We write that number $P (y = 1 ∣ X)$ .

WHY. The vertical bar $∣$ reads "given". So $P (y = 1 ∣ X)$ is "the probability the colour is leopard, given we can see where the dot is". This is the posterior — the answer everyone ultimately wants, whether they call themselves generative or discriminative.

PICTURE. The grey dot sits in the overlap. Around it we draw a small window; inside that window we count what fraction of nearby dots are lavender. That fraction is $P (y = 1 ∣ X)$ .

probability P (it’s a leopard y = 1 ∣ we saw this position X)

Each piece: $P$ = "how likely", $y = 1$ = the outcome we're asking about, $X$ = the fact we already know.

Step 3 — Two honest ways to fill in that number

WHAT. There are exactly two strategies, and this is the fork the whole parent note is about.

Discriminative: stare only at the boundary between the clouds and learn "left of this curve → coral, right → lavender." Answer $P (y = 1 ∣ X)$ directly.
Generative: learn each cloud's shape separately — where lavender dots pile up, where coral dots pile up — then reason backwards to the colour.

WHY. They chase the same target ( $P (y ∣ X)$ ) from opposite ends. The discriminative one ignores everything except the frontier; the generative one models the whole landscape and gets the frontier as a by-product.

PICTURE. Same dots, two overlays: a single dashed boundary curve (discriminative), versus two shaded hills describing each cloud (generative).

Step 4 — The generative pieces, one hill at a time

WHAT. To describe a cloud we need two things:

$P (X ∣ y = 1)$ = the likelihood: "if the dot is a leopard, how likely is this position?" This is the height of the lavender hill at $X$ .
$P (y = 1)$ = the prior: "before looking at the position, what fraction of all dots are leopards?" This is just how tall the lavender hill is overall — its total mass.

WHY. A tall narrow hill and a short wide hill can have the same mass. Likelihood is the shape; prior is the amount. To weigh a colour fairly at a point we need both — a common colour that also fits the position well is the strong candidate.

PICTURE. Two hills seen side-on. The lavender hill's height at the grey dot's x-position is the likelihood; the shaded area under each hill is its prior.

height of lavender hill at X P (X ∣ y = 1) and total lavender mass (prior) P (y = 1)

Step 5 — Multiply: weighting each hill at the dot

WHAT. For the grey dot, compute for each colour:

score (y) = how well the dot fits this hill P (X ∣ y) \times how common this colour is P (y)

WHY multiply? Both conditions must hold. A dot is a strong leopard candidate only if the position fits the leopard hill AND leopards are reasonably common. "AND" of two probabilities is a product. If either factor is tiny, the score collapses — exactly what we want.

PICTURE. At the grey dot's x-position we drop a vertical line; it pierces the lavender hill at one height and the coral hill at another. Each height is then scaled by that colour's mass. Two bars result: a lavender score and a coral score.

Step 6 — Normalise: turning two scores into a probability

WHAT. The two scores don't add to 1, so they aren't yet probabilities. Divide each by their sum:

P (y = 1 ∣ X) = \frac{P ( X ∣ 1 ) P ( 1 ) lavender score}{lavender score P ( X ∣ 1 ) P ( 1 ) + coral score P ( X ∣ 0 ) P ( 0 )}

WHY divide by the sum? A probability over two colours must total 1. The sum in the denominator is the evidence $P (X)$ — "how likely is a dot at this position from any colour at all." Dividing shares the whole "1" out between the colours in proportion to their scores.

PICTURE. The two bars from Step 5 stacked into one full bar of height 1; the lavender fraction of that bar is $P (y = 1 ∣ X)$ .

evidence P (X) = y^{'} \sum P (X ∣ y^{'}) P (y^{'}) ⟸ sum over every colour y^{'}

With our numbers: $P (y = 1 ∣ X) = \frac{0.18}{0.18 + 0.14} = \frac{0.18}{0.32} = 0.5625$ . Just above half — leopard, barely. This whole line is Bayes' theorem in disguise.

Step 7 — Edge & degenerate cases (never let the reader trip)

WHAT. Four ways the picture can go strange.

PICTURE. Four mini-panels, each a warning.

Common mistake The dangerous corners

(a) Dot far from both hills. Both likelihoods $\approx 0$ , so the denominator $P (X) \approx 0$ — division by nearly zero. The ratio can still be fine, but the model is guessing in empty space. A discriminative judge just extends its line and answers confidently anyway — sometimes wrongly.

(b) One colour missing (prior $= 0$ ). If $P (y = 1) = 0$ , its score is $0 \times (anything) = 0$ ; that colour can never be chosen, no matter how well the position fits. Priors gate everything.

(c) Perfectly separated clouds (no overlap). In the gap, one score is essentially 0 and the other positive, so $P (y ∣ X)$ snaps to a clean 0 or 1. Both approaches agree here — the easy case.

(d) Identical hills, different priors. If $P (X ∣ 1) = P (X ∣ 0)$ everywhere, the position tells you nothing; the answer collapses to the prior alone: $P (y = 1 ∣ X) = P (y = 1)$ . Position is ignored — as it should be.

The one-picture summary

WHAT. The entire journey on a single canvas: two hills → drop a line at the new dot → read two heights → weight by mass → normalise into one bar → read off the leopard fraction.

Recall Feynman retelling (say it in plain words)

I scatter every animal I've seen as a coloured dot on paper — lavender leopards, coral cats. A new grey dot arrives and I want to know its colour.

The generative way: I first draw a smooth hill over each colour's cloud — that hill tells me how likely a dot of that colour is to land at any spot. At the grey dot I measure how high each hill is (does the position fit that colour?), and I multiply by how common that colour is overall (its total mass). That gives one score per colour. The scores don't add to 1, so I divide each by their total. The lavender share of the total is my probability it's a leopard. That division-by-the-total is Bayes' theorem, and the total itself is the "evidence" — how likely the position is under any colour.

The discriminative way skips the hills entirely and just learns the dividing curve, answering the colour question directly. Same target, less work, but it can never draw a new animal — it only ever judges.

Edge cases keep me honest: empty space makes the total tiny, a zero prior kills a colour outright, non-overlapping clouds give crisp 0/1 answers, and identical hills mean I fall back on which colour is simply more common.

Recall Quick self-check

Why do we multiply likelihood by prior instead of adding? ::: Both conditions must hold at once (good fit AND common colour); the joint probability of an "AND" is a product, and either factor being tiny should kill the score — addition wouldn't do that. What does the denominator $P (X)$ mean geometrically? ::: The total height of all hills stacked at the dot's position — how likely that position is under any colour — used to rescale the scores so they sum to 1. What happens when the two hills are identical? ::: Position carries no information, so $P (y ∣ X)$ reduces to the prior $P (y)$ alone.

Connections

4.5.01 Generative vs discriminative models (Hinglish) — parent topic
4.1.01-Bayes-Theorem — the normalise step is this theorem
4.5.04-Naive-Bayes-Classifier — builds each hill assuming independent features
3.3.01-Logistic-Regression — models the dashed boundary directly
4.5.05-Gaussian-Mixture-Models — hills as sums of Gaussians
4.5.02-Variational-Autoencoders / 4.5.03-Generative-Adversarial-Networks — deep ways to draw the hills
4.6.01-Semi-Supervised-Learning — where knowing the hills pays off