Generative vs discriminative models — Hinglish notes

4.5.1 · AI-ML › Generative Models

YEH kyun matter karta hai: Discriminative models ka jawab hota hai "kaun si class?". Generative models ka jawab hota hai "yeh data kaise create hua?" – jo inhe superpowers deta hai jaise data synthesis, anomaly detection, aur missing features handle karna.

Mathematical Foundation

Probability ki Kahani

Connection Derive Karna (First Principles Se)

Bayes' theorem se kyun shuru karein? Kyunki yeh generative aur discriminative approaches ke beech ka bridge hai.

Bayes' theorem kehta hai: $P (y ∣ X) = \frac{P ( X ∣ y ) \cdot P ( y )}{P ( X )}$

Har term kyun?

$P (X ∣ y)$ likelihood – "agar yeh cat hai, toh hum kaun se features dekhenge?"
$P (y)$ = prior – "generally cats kitni common hain?"
$P (X)$ = evidence – "yeh feature combination overall kitni likely hai?"
$P (y ∣ X)$ = posterior – "in features ko dekh ke, class kya hai?"

Key Insight: $P (X) = \sum_{y^{'}} P (X ∣ y^{'}) \cdot P (y^{'})$

Yeh sum kyun? Kyunki data $X$ KISI BHI class se aa sakta hai, isliye hum sab possibilities pe marginalize (sum) karte hain.

Isse yeh milta hai: $P (y ∣ X) = \frac{P ( X ∣ y ) \cdot P ( y )}{\sum _{y^{'}} P ( X ∣ y ^{'} ) \cdot P ( y ^{'} )}$

Discriminative approach: $P (y ∣ X)$ ko directly ek function $f (X) \to y$ se model karo.

Generative approach: $P (X ∣ y)$ aur $P (y)$ ko alag-alag model karo, phir Bayes use karke $P (y ∣ X)$ compute karo.

Figure — Generative vs discriminative models

Detailed Examples

Worked example Example 1: Email Spam Classification (Binary Case)

Setup: Emails ko spam ( $y = 1$ ) ya not spam ( $y = 0$ ) classify karo word counts $X = [x_{1}, x_{2}, .., x_{n}]$ ke basis pe.

Discriminative Approach (Logistic Regression): $P (y = 1∣ X)$ directly model karo: $P (y = 1∣ X) = σ (w^{T} X + b) = \frac{1}{1 + e ^{- (w^{T} X + b)}}$

Yeh step kyun? Hum decision boundary ko directly parameterize karte hain. Model seekhta hai: "Agar 'lottery' 5 baar aaya aur 'meeting' 0 baar, toh spam ki probability = 0.95."

Training: Training data pe $\prod_{i} P (y_{i} ∣ X_{i})$ maximize karo.

Generative Approach (Naive Bayes): $P (X ∣ y)$ aur $P (y)$ model karo: $P (X ∣ y) = \prod_{j = 1}^{n} P (x_{j} ∣ y)$

Yeh product kyun? Naive Bayes assume karta hai ki features class diye jaane pe independent hain (naive assumption, lekin aksar kaam karta hai).

Word counts ke liye: $P (x_{j} ∣ y = 1) = \frac{count of word j in spam emails}{total words in spam emails}$

Count ratios kyun? Yeh estimate karta hai "spam generate karte waqt YEH word kitni baar aata hai?"

Phir prior compute karo: $P (y = 1) = \frac{number of spam emails}{total emails}$

Classification: Naye email $X_{n e w}$ ke liye: $P (y = 1∣ X_{n e w}) = \frac{P ( X _{n e w} ∣ y = 1 ) \cdot P ( y = 1 )}{P ( X _{n e w} ∣ y = 1 ) \cdot P ( y = 1 ) + P ( X _{n e w} ∣ y = 0 ) \cdot P ( y = 0 )}$

Is sum se divide kyun karein? Normalization ensure karta hai ki probabilities ka sum 1 ho.

Worked example Example 2: Image Classification (MNIST Digits)

Discriminative (Neural Network): $P (y = k ∣ X) = softmax (f_{θ} (X))_{k} = \frac{e ^{z_{k}}}{\sum _{j = 1}^{10} e ^{z_{j}}}$ jahan $z = f_{θ} (X)$ network ka output hai.

Softmax kyun? Arbitrary scores ko valid probabilities mein convert karta hai (positive, sum to 1).

Training: Cross-entropy loss minimize karo: $L = - \sum_{i} lo g P (y_{i} ∣ X_{i})$

Negative log kyun? Probability maximize karna = negative log-probability minimize karna.

Generative (Gaussian Naive Bayes ya VAE):

Gaussian Naive Bayes ke liye, har pixel $j$ ko aise model karo: $P (x_{j} ∣ y = k) = N (μ_{j k}, σ_{j k}^{2})$

Gaussian kyun? Pixels continuous hain, isliye hum inhe continuous distributions se model karte hain.

$μ_{j k}$ seekho = digit $k$ ke liye position $j$ pe mean pixel value.

Full likelihood: $P (X ∣ y = k) = \prod_{j = 1}^{784} \frac{1}{2 π σ _{j k}^{2}} exp (- \frac{( x _{j} - μ _{j k} ) ^{2}}{2 σ _{j k}^{2}})$

Classification: Wo class chuno jiska $P (y = k) \cdot P (X ∣ y = k)$ sabse zyada ho.

Bonus: Naye digit samples generate karo $X \sim P (X ∣ y = 5)$ se sample karke nayi "5"s banao.

Comparison Table

Aspect	Discriminative	Generative
Models	$P (y ∥ X)$	$P (X ∥ y)$ aur $P (y)$
Goal	Decision boundary	Data distribution
Examples	Logistic Regression, SVM, Neural Nets	Naive Bayes, GM, VAE, GAN
Data Efficiency	Zyada efficient (kam params)	Zyada data chahiye
Can Generate	❌ No	✅ Yes
Missing Features	Struggle karta hai	Marginalization se handle karta hai
Asymptotic Accuracy	Higher (boundary pe focus)	Lower (sab kuch model karta hai)

Discriminative aksar kyun jeetta hai? Yeh sirf decision boundary model karta hai, poora data distribution nahi. Kam parameters = kam data chahiye = zyada data hone pe better generalization.

Generative powerful kyun hai? Yeh data ki structure samajhta hai, synthesis, semi-supervised learning, aur robustness enable karta hai.

Common mistake Common Misconceptions

Galti 1: "Generative models hamesha better hote hain kyunki yeh zyada model karte hain." Kyun sahi lagta hai: Zyada modeling zyada intelligent lagti hai. Fix: Discriminative models classification ke liye zyada sample-efficient hote hain. Agar tumhare paas enough data hai aur SIRF classification chahiye, toh discriminative jeetta hai. Generative tab shine karta hai jab tumhe generate karna ho, missing data handle karna ho, ya unlabeled data use karna ho.

Galti 2: "Discriminative models se data generate nahi kar sakte." Kyun sahi lagta hai: Yeh $P (X)$ model nahi karte. Fix: SACH hai, lekin tum conditional discriminative models use kar sakte ho (jaise conditional GANs jahan discriminator discriminative hai, lekin generator generative hai). Hybrid approaches exist karte hain.

Galti 3: "Bayes theorem sirf generative models ke liye hai." Kyun sahi lagta hai: Derivation explicitly $P (X ∣ y)$ use karta hai. Fix: Bayes theorem ek mathematical identity hai. Discriminative models isse implicitly use karte hain $P (y ∣ X)$ directly model karke, intermediate steps skip karke. Dono approaches valid hain.

Galti 4: "Naive Bayes assume karta hai ki features independent hain." Kyun sahi lagta hai: Naam mein "naive" hai. Fix: Yeh independence class diye jaane pe assume karta hai ( $P (X ∣ y) = \prod P (x_{i} ∣ y)$ ), unconditional independence nahi. Features classes ke across correlated ho sakte hain, lekin model within-class correlation ignore karta hai. Yeh simplification aksar surprisingly well kaam karti hai (80/20 principle).

Recall 12-Saal-Ke-Bacche Ko Explain Karo

Socho tum photos mein dogs aur cats pehchanna seekh rahe ho.

Discriminative tarika: Tum rules seekhte ho jaise "agar iske pointed ears aur whiskers hain, toh shayad cat hai." Tum SIRF inhe alag karne ki parwah karte ho. Tum realistic cat ya dog draw nahi kar sakte, lekin tum haan/na ke decision mein BAHUT acche ho.

Generative tarika: Tum cats aur dogs ke baare mein SAB KUCH study karte ho – inke fur kaise badhte hain, ear shapes, typical colors. Ab tum scratch se realistic cats aur dogs DRAW kar sakte ho. Tum inhe identify bhi kar sakte ho (apni drawing knowledge compare karke), lekin seekhne mein zyada mehnat lagti hai.

Toh discriminative = identification ke liye jaldi seekhna. Generative = deeper understanding, naye examples create kar sakta hai.

Connections

4.5.02-Variational-Autoencoders – Deep generative model ka example
4.5.03-Generative-Adversarial-Networks – Hybrid generative approach
4.1.01-Bayes-Theorem – Mathematical foundation
4.5.04-Naive-Bayes-Classifier – Classic generative classifier
3.3.01-Logistic-Regression – Classic discriminative classifier
4.5.05-Gaussian-Mixture-Models – Generative clustering
4.6.01-Semi-Supervised-Learning – Generative models yahan excel karte hain

#flashcards/ai-ml

Generative aur discriminative models ke beech key difference kya hai? :: Discriminative models $P (y ∣ X)$ (decision boundaries) seekhte hain, jabki generative models $P (X ∣ y)$ aur $P (y)$ (data kaise generate hota hai) seekhte hain.

Classification ke liye Bayes' theorem likhо :: $P (y ∣ X) = \frac{P ( X ∣ y ) \cdot P ( y )}{P ( X )} = \frac{P ( X ∣ y ) \cdot P ( y )}{\sum _{y^{'}} P ( X ∣ y ^{'} ) \cdot P ( y ^{'} )}$

Teen generative models ke naam batao

Naive Bayes, Gaussian Mixture Models (GMM), Variational Autoencoders (VAE), GANs

Teen discriminative models ke naam batao

Logistic Regression, Support Vector Machines (SVM), Neural Networks, Decision Trees

Generative model mein prior

P (y)

kya hai?

Class

y

ki probability koi bhi features dekhne se pehle (jaise spam emails ki base rate)

Likelihood

P (X ∣ y)

kya hai?

Features

X

observe karne ki probability diya gaya ki true class

y

hai (class

y

data kaise generate karta hai)

Discriminative models ko typically kam data kyun chahiye?

Yeh sirf decision boundary model karte hain, poora data distribution nahi, isliye kam parameters chahiye

Generative models ka discriminative ke mukable ek advantage kya hai?

Naya synthetic data generate kar sakte hain, missing features handle kar sakte hain, semi-supervised learning perform kar sakte hain

Naive Bayes mein "naive" kya assume karta hai?

Features class diye jaane pe conditionally independent hain:

P (X ∣ y) = \prod_{i} P (x_{i} ∣ y)

Spam classification mein generative model kya seekhta hai?

P (words ∣ spam)

aur

P (words ∣ not spam)

– har class ke liye word distributions

Generative model se classify kaise karte hain?

Har class ke liye

P (y ∣ X) \propto P (X ∣ y) \cdot P (y)

compute karo aur sabse zyada wala chuno

P (X)

ke context mein marginalization kya hai?

Sab possible classes pe sum karna:

P (X) = \sum_{y} P (X ∣ y) P (y)

Discriminative neural networks mein softmax kyun aata hai?

Raw scores (logits) ko valid probabilities mein convert karta hai jo sum to 1 karte hain

Agar tum sirf classification accuracy chahte ho bade data ke saath, toh kaun sa better hai?

Discriminative models (zyada sample-efficient, asymptotically zyada accurate)

Ek aisa scenario batao jahan generative models excel karte hain :: Jab tumhe naye samples generate karne hon, density estimation karni ho, ya unlabeled data leverage karna ho