Foundations — Generative vs discriminative models

Before you can read a single equation on the parent note, you need to own every symbol in it. This page introduces them one at a time, from absolute zero, each anchored to a picture. Nothing here assumes you have seen probability notation before.

1. Features and labels: $X$ and $y$

The picture. Think of one row of a spreadsheet. The many columns on the left are $X$ (the clues). The single column on the right is $y$ (the truth).

Why the topic needs it. The whole debate — generative vs discriminative — is about which direction of the arrow between $X$ and $y$ you choose to learn. So we must name both ends of that arrow first.

2. What "probability" $P(\cdot )$ actually means

The picture. Picture a bar that can only be filled between empty and full. $P=0.95$ is a bar filled 95% of the way up.

3. Conditional probability: the vertical bar $P(A\mid B)$

This is the single most important notation on the parent page, so we build it slowly.

The picture. Imagine a big rectangle = all emails. Shade the region "spam" and the region "contains the word lottery". $P(\text{spam}\mid \text{lottery})$ ignores the whole rectangle and looks only inside the lottery region, then asks: what fraction of that slice is spam?

Why the topic needs it. The parent note's entire distinction is written with this bar:

• Discriminative learns $P(y\mid X)$ — "given the clues, which answer?"
• Generative learns $P(X\mid y)$ — "given the answer, which clues would we see?"

These are the two directions of the same bar.

4. The four named quantities in Bayes' theorem

The parent note names $P(X\mid y)$, $P(y)$, $P(X)$, and $P(y\mid X)$. Here is each, from zero.

The picture. Three levers feed one dial. The prior and the likelihood are levers you learn from data; the evidence is a normaliser; the posterior is the dial they move.

5. Summation $\sum$ and marginalising

The parent note writes the evidence as $P(X)={\sum }_{y^{\prime }}P(X\mid y^{\prime })P(y^{\prime })$. Two new symbols: $\sum$ and $y^{\prime }$.

The picture. If there are only two classes, spam and not-spam, then ${\sum }_{y^{\prime }}P(X\mid y^{\prime })P(y^{\prime })=P(X\mid \text{spam})P(\text{spam})+P(X\mid \text{not})P(\text{not}).$ Two terms, added. That's all $\sum$ did here.

6. Product $\prod$ and the independence assumption

Naive Bayes on the parent page uses $P(X\mid y)={\prod }_{j=1}^{n}P(x_j\mid y)$.

7. The exponential $e^{(\cdot )}$, $\sigma$, softmax, and the Gaussian

These appear inside the parent's examples. You only need to recognise their shape.

8. Argmax and $\propto$: picking the winner

The parent note says "pick the class with highest $P(y=k)P(X\mid y=k)$."

How these foundations feed the topic

Equipment checklist

Test yourself — cover the right side and answer out loud.

Connections

• Parent topic — the note these symbols serve
• 4.1.01-Bayes-Theorem — the bridge equation built from Sections 3–5
• 4.5.04-Naive-Bayes-Classifier — uses the product/independence of Section 6
• 3.3.01-Logistic-Regression — uses the sigmoid of Section 7
• 4.5.05-Gaussian-Mixture-Models — uses the Gaussian of Section 7
• 4.5.02-Variational-Autoencoders — deep generative model built on these ideas
• 4.5.03-Generative-Adversarial-Networks — hybrid generative approach
• 4.6.01-Semi-Supervised-Learning — where generative models shine