Generative vs discriminative models

WHY this matters: Discriminative models answer "which class?". Generative models answer "how is this data created?" – giving them superpowers like data synthesis, anomaly detection, and handling missing features.

Mathematical Foundation

The Probability Story

Deriving the Connection (From First Principles)

WHY start with Bayes' theorem? Because it's the bridge between generative and discriminative approaches.

Bayes' theorem states: $P(y|X) = \frac{P(X|y) \cdot P(y)}{P(X)}$

WHY each term?

$P(X|y)$ likelihood – "if it's a cat, what features would we see?"
$P(y)$ = prior – "how common are cats in general?"
$P(X)$ = evidence – "how likely is this feature combination overall?"
$P(y|X)$ = posterior – "given these features, what class is it?"

The Key Insight: $P(X) = \sum_{y'} P(X|y') \cdot P(y')$

WHY this sum? Because the data $X$ could come from ANY class, so we marginalize (sum) over all possibilities.

This gives us: $P(y|X) = \frac{P(X|y) \cdot P(y)}{\sum_{y'} P(X|y') \cdot P(y')}$

Discriminative approach: Model $P(y|X)$ directly with a function $f(X) \to y$ .

Generative approach: Model $P(X|y)$ and $P(y)$ separately, then compute $P(y|X)$ using Bayes.

Figure — Generative vs discriminative models

Detailed Examples

Worked example Example 1: Email Spam Classification (Binary Case)

Setup: Classify emails as spam ( $y=1$ ) or not spam ( $y=0$ ) based on word counts $X = [x_1, x_2, .., x_n]$ .

Discriminative Approach (Logistic Regression): Model $P(y=1|X)$ directly: $P(y=1|X) = \sigma(w^T X + b) = \frac{1}{1 + e^{-(w^T X + b)}}$

WHY this step? We parameterize the decision boundary directly. The model learns: "If 'lottery' appears 5 times and 'meeting' appears 0 times, probability of spam = 0.95."

Training: Maximize $\prod_i P(y_i|X_i)$ over training data.

Generative Approach (Naive Bayes): Model $P(X|y)$ and $P(y)$ : $P(X|y) = \prod_{j=1}^{n} P(x_j|y)$

WHY this product? Naive Bayes assumes features are independent given the class (naive assumption, but often works).

For word counts: $P(x_j|y=1) = \frac{\text{count of word } j \text{ in spam emails}}{\text{total words in spam emails}}$

WHY count ratios? This estimates "how often does THIS word appear when generating spam?"

Then compute prior: $P(y=1) = \frac{\text{number of spam emails}}{\text{total emails}}$

Classification: For new email $X_{new}$ : $P(y=1|X_{new}) = \frac{P(X_{new}|y=1) \cdot P(y=1)}{P(X_{new}|y=1) \cdot P(y=1) + P(X_{new}|y=0) \cdot P(y=0)}$

WHY divide by this sum? Normalization ensures probabilities sum to 1.

Worked example Example 2: Image Classification (MNIST Digits)

Discriminative (Neural Network): $P(y=k|X) = \text{softmax}(f_\theta(X))_k = \frac{e^{z_k}}{\sum_{j=1}^{10} e^{z_j}}$ where $z = f_\theta(X)$ is the network output.

WHY softmax? Converts arbitrary scores into valid probabilities (positive, sum to 1).

Training: Minimize cross-entropy loss: $L = -\sum_i \log P(y_i|X_i)$

WHY negative log? Maximizing probability = minimizing negative log-probability.

Generative (Gaussian Naive Bayes or VAE):

For Gaussian Naive Bayes, model each pixel $j$ as: $P(x_j|y=k) = \mathcal{N}(\mu_{jk}, \sigma_{jk}^2)$

WHY Gaussian? Pixels are continuous, so we model them with continuous distributions.

Learn $\mu_{jk}$ = mean pixel value at position $j$ for digit $k$ .

Full likelihood: $P(X|y=k) = \prod_{j=1}^{784} \frac{1}{\sqrt{2\pi\sigma_{jk}^2}} \exp\left(-\frac{(x_j - \mu_{jk})^2}{2\sigma_{jk}^2}\right)$

Classification: Pick class with highest $P(y=k) \cdot P(X|y=k)$ .

Bonus: Generate new digit samples by sampling $X \sim P(X|y=5)$ to create new "5"s.

Comparison Table

Aspect	Discriminative	Generative
Models	$P(y\\|X)$	$P(X\\|y)$ and $P(y)$
Goal	Decision boundary	Data distribution
Examples	Logistic Regression, SVM, Neural Nets	Naive Bayes, GM, VAE, GAN
Data Efficiency	More efficient (fewer params)	Needs more data
Can Generate	❌ No	✅ Yes
Missing Features	Struggles	Handles via marginalization
Asymptotic Accuracy	Higher (focuses on boundary)	Lower (models everything)

WHY discriminative often wins? It only models the decision boundary, not the entire data distribution. Fewer parameters = less data needed = better generalization when data is abundant.

WHY generative is powerful? It understands the data's structure, enabling synthesis, semi-supervised learning, and robustness.

Common mistake Common Misconceptions

Mistake 1: "Generative models are always better because they model more." Why it feels right: More modeling seems more intelligent. The fix: Discriminative models are more sample-efficient for classification. If you have enough data and ONLY care about classification, discriminative wins. Generative shines when you need to generate, handle missing data, or leverage unlabeled data.

Mistake 2: "You can't use discriminative models to generate data." Why it feels right: They don't model $P(X)$ . The fix: TRUE, but you can use conditional discriminative models (like conditional GANs where the discriminator is discriminative, but the generator is generative). Hybrid approaches exist.

Mistake 3: "Bayes theorem is only for generative models." Why it feels right: The derivation explicitly uses $P(X|y)$ . The fix: Bayes theorem is a mathematical identity. Discriminative models implicitly use it by modeling $P(y|X)$ directly, skipping the intermediate steps. Both approaches are valid.

Mistake 4: "Naive Bayes assumes features are independent." Why it feels right: The name says "naive." The fix: It assumes independence given the class ( $P(X|y) = \prod P(x_i|y)$ ), not unconditional independence. Features can be corelated across classes, but the model ignores within-class correlation. This simplification often works surprisingly well (80/20 principle).

Recall Explain to a 12-Year-Old

Imagine you're learning to tell apart dogs and cats in photos.

Discriminative way: You learn rules like "if it has pointy ears and whiskers, it's probably a cat." You ONLY care about telling them apart. You can't draw a realistic cat or dog, but you're REALLY good at the yes/no decision.

Generative way: You study EVERYTHING about how cats and dogs look – how their fur grows, ear shapes, typical colors. Now you can DRAW realistic cats and dogs from scratch. You can also identify them (by comparing your drawing knowledge), but it takes more effort to learn.

So discriminative = faster learner for identification. Generative = deeper understanding, can create new examples.

Connections

4.5.02-Variational-Autoencoders – Example of deep generative model
4.5.03-Generative-Adversarial-Networks – Hybrid generative approach
4.1.01-Bayes-Theorem – Mathematical foundation
4.5.04-Naive-Bayes-Classifier – Classic generative classifier
3.3.01-Logistic-Regression – Classic discriminative classifier
4.5.05-Gaussian-Mixture-Models – Generative clustering
4.6.01-Semi-Supervised-Learning – Generative models excel here

#flashcards/ai-ml

What is the key difference between generative and discriminative models? :: Discriminative models learn $P(y|X)$ (decision boundaries), while generative models learn $P(X|y)$ and $P(y)$ (how data is generated).

Write Bayes' theorem for classification :: $P(y|X) = \frac{P(X|y) \cdot P(y)}{P(X)} = \frac{P(X|y) \cdot P(y)}{\sum_{y'} P(X|y') \cdot P(y')}$

Name three generative models

Naive Bayes, Gaussian Mixture Models (GMM), Variational Autoencoders (VAE), GANs

Name three discriminative models

Logistic Regression, Support Vector Machines (SVM), Neural Networks, Decision Trees

What is the prior

P(y)

in a generative model?

The probability of class

y

before seeing any features (e.g., base rate of spam emails)

What is the likelihood

P(X|y)

The probability of observing features

X

given that the true class is

y

(how class

y

generates data)

Why do discriminative models typically need less data?

They model only the decision boundary, not the full data distribution, requiring fewer parameters

What is one advantage of generative models over discriminative?

Can generate new synthetic data, handle missing features, perform semi-supervised learning

What does the "naive" in Naive Bayes assume?

Features are conditionally independent given the class:

P(X|y) = \prod_i P(x_i|y)

In spam classification, what does a generative model learn?

P(\text{words}|\text{spam})

and

P(\text{words}|\text{not spam})

– the word distributions for each class

How do you classify with a generative model?

Compute

P(y|X) \propto P(X|y) \cdot P(y)

for each class and pick the highest

What is marginalization in the context of

P(X)

Summing over all possible classes:

P(X) = \sum_y P(X|y) P(y)

Why does softmax appear in discriminative neural networks?

Converts raw scores (logits) into valid probabilities that sum to 1

If you only care about classification accuracy with large data, which is better?

Discriminative models (more sample-efficient, asymptotically more accurate)

Give one scenario where generative models excel :: When you need to generate new samples, perform density estimation, or leverage unlabeled data

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Samjho ki tumhe photos mein se cats aur dogs ko identify karna hai. Do tarike hain.

Discriminative model (jaise Logistic Regression ya Neural Network) sirf yeh seekhta hai ki "boundary kahan hai" –agar pointy ears hain aur ch

Mathematical Foundation

The Probability Story

Deriving the Connection (From First Principles)

Detailed Examples

Comparison Table

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Generative Models

Connections