Evaluating generative models (FID, IS) — Hinglish notes

4.5.17 · AI-ML › Generative Models

Overview

Evaluating generative models discriminative tasks se fundamentally alag hai kyunki koi ek "correct" output nahi hota—hum quality (realism) aur diversity (data distribution ki coverage) ki parwah karte hain. Traditional metrics jaise accuracy yahan apply nahi hoti, isliye hum specialized scores use karte hain: Inception Score (IS) aur Fréchet Inception Distance (FID).

Figure — Evaluating generative models (FID, IS)

Inception Score (IS)

Core Idea

Inception Score quality aur diversity measure karta hai yeh check karke:

Sharpness (quality): Har generated image confidently ek category mein classify honi chahiye → low entropy conditional distribution $p (y ∣ x)$
Diversity: Overall label distribution uniform honi chahiye → high entropy marginal $p (y)$

First Principles se Derivation

Step 1: Ek "good" generative model kya hota hai?

Ek achha generator produce karna chahiye:

Meaningful images: Agar aap ek classifier ko ek image dikhao, toh woh confident hona chahiye (jaise "90% dog, 10% other") → $p (y ∣ x)$ ki low entropy hoti hai
Diverse images: Sabhi images mein milake, aapko sabhi classes dikhni chahiye → $p (y)$ ki high entropy hoti hai

Step 2: KL divergence ise quantify karta hai

$D_{KL} (p (y ∣ x) ∣∣ p (y)) = \sum_{y} p (y ∣ x) lo g \frac{p ( y ∣ x )}{p ( y )}$

Yeh step kyun? KL divergence tab bada hota hai jab:

$p (y ∣ x)$ peaked ho (low entropy) → achhi quality
$p (y)$ uniform ho (high entropy) → achhi diversity

Step 3: Sabhi generated samples par average karo

$KL_{avg} = E_{x \sim p_{g}} [\sum_{y} p (y ∣ x) lo g \frac{p ( y ∣ x )}{p ( y )}]$

Hum ise logarithm properties se split karte hain:

$= E_{x \sim p_{g}} [\sum_{y} p (y ∣ x) lo g p (y ∣ x)] - E_{x \sim p_{g}} [\sum_{y} p (y ∣ x) lo g p (y)]$

Pehla term $- H (y ∣ x)$ hai (negative conditional entropy), doosra simplify hota hai kyunki $E_{x} [\sum_{y} p (y ∣ x) lo g p (y)] = \sum_{y} p (y) lo g p (y) = - H (y)$ .

Toh: $KL_{avg} = H (y) - H (y ∣ x)$

Zyada $H (y)$ (diverse labels) aur kam $H (y ∣ x)$ (confident predictions) → zyada IS.

Step 4: Interpretability ke liye exponentiate karo

$IS = exp (KL_{avg})$

Exponentiate kyun? Log-space se ek aisi scale par convert karne ke liye jahan higher = better, aur "effective number of classes" wali intuition se match karne ke liye. IS ≈ 1 matlab terrible (sabhi images ek jaisi lagte hain ya noise hain), IS ≈ number of classes matlab perfect.

Worked Example

Worked example Toy Data ke liye IS Calculate Karna

Maano hum 3 images generate karte hain jinke Inception predictions hain:

Image 1: $p (y ∣ x_{1}) = [0.9, 0.05]$ (dog)
Image 2: $p (y ∣ x_{2}) = [0.05, 0.9, 0.05]$ (cat)
Image 3: $p (y ∣ x_{3}) = [0.05, 0.05, 0.9]$ (car)

Step 1: Marginal $p (y)$ compute karo $p (y) = \frac{1}{3} ([0.9, 0.05] + [0.05, 0.9, 0.05] + [0.05, 0.05, 0.9]) = [0.333, 0.333, 0.333]$ Uniform kyun? Kyunki humne har class ke liye ek confident sample generate kiya.

Step 2: Har image ke liye KL (image 1 dikha rahe hain): $D_{KL} (p (y ∣ x_{1}) ∣∣ p (y)) = 0.9 lo g \frac{0.9}{0.333} + 0.05 lo g \frac{0.05}{0.333} + 0.05 lo g \frac{0.05}{0.333}$ $= 0.9 (0.995) + 0.05 (- 1.897) + 0.05 (- 1.897) = 0.895 - 0.190 = 0.705$ Symmetry se, images 2 aur 3 bhi 0.705 dete hain. Step 3: Average karo aur exponentiate karo $IS = exp (\frac{0.705 + 0.705 + 0.705}{3}) = exp (0.705) \approx 2.02$

Interpretation: IS ≈ 2 reasonable hai 3-class generation ke liye jab clear mode coverage ho.

Common mistake Common Mistake: "Higher IS Matlab Hamesha Better Model"

Galat idea: IS = 15 wala model IS = 10 wale se better hai.

Yeh sahi kyun lagta hai: Higher score = better, jaise accuracy.

Problem: IS mein critical flaws hain:

Mode collapse invisible hai: Agar aapka GAN sirf perfect dogs generate karta hai (ek mode), toh IS high rahega kyunki $p (y ∣ x)$ confident hai aur $p (y)$ "dog" class par collapse ho jata hai → phir bhi high KL.
Realism ignore karta hai: IS sirf check karta hai ki images classifiable hain ya nahi, realistic lagte hain ya nahi yeh nahi. Ek sharp lekin nonsensical image achha score kar sakti hai.
Dataset-dependent: ImageNet par IS ≠ CIFAR-10 par IS. Koi absolute scale nahi. Fix: IS ko hamesha FID ke saath use karo. FID mode collapse pakad leta hai real data distribution se compare karke.

Fréchet Inception Distance (FID)

Core Idea

FID real images aur generated images ke distributions ke beech Inception-v3 ke feature space mein distance measure karta hai. Lower FID = distributions zyada close hain = better generation.

First Principles se Derivation

Step 1: Gaussians ke liye Fréchet distance

Agar do distributions Gaussian hain: $p_{r} \sim N (μ_{r}, Σ_{r}), p_{g} \sim N (μ_{g}, Σ_{g})$

Fréchet distance (Gaussians ke liye Wasserstein-2 distance bhi kaha jata hai) ka ek closed form hota hai:

$d_{F}^{2} (N (μ_{1}, Σ_{1}), N (μ_{2}, Σ_{2})) = ∣ μ_{1} - μ_{2} ∣ ∣^{2} + Tr (Σ_{1} + Σ_{2} - 2 (Σ_{1} Σ_{2})^{1/2})$

Yeh formula kyun? Yeh 2-Wasserstein distance se aata hai:

$W_{2} (p, q) = in f_{π} E_{(x, y) \sim π} [∣∣ x - y ∣ ∣^{2}]$

jahan $π$ distributions ke beech ek coupling hai. Gaussians ke liye, is infimum ka analytical solution hota hai.

Step 2: Feature distributions par apply karo

Features extract karo:

$N$ real images ke liye: ${f_{r}^{(i)}}_{i = 1}^{N}$ Inception pool3 layer se (2048-dim)
$N$ fake images ke liye: ${f_{g}^{(i)}}_{i = 1}^{N}$

Parameters estimate karo: $μ_{r} = \frac{1}{N} \sum_{i} f_{r}^{(i)}, Σ_{r} = \frac{1}{N - 1} \sum_{i} (f_{r}^{(i)} - μ_{r}) (f_{r}^{(i)} - μ_{r})^{T}$

(Same $μ_{g}, Σ_{g}$ ke liye)

Step 3: Matrix square root compute karo

Term $(Σ_{r} Σ_{g})^{1/2}$ eigenvalue decomposition se compute hota hai:

Agar $Σ_{r} Σ_{g} = U Λ U^{T}$ , toh $(Σ_{r} Σ_{g})^{1/2} = U Λ^{1/2} U^{T}$

Kyun? Kyunki positive semi-definite matrices ke liye, square root well-defined hoti hai.

Step 4: Difference ka trace

$Tr (Σ_{r} + Σ_{g} - 2 (Σ_{r} Σ_{g})^{1/2})$

Trace kyun? Yeh sabhi 2048 dimensions mein covariance mismatch ko sum karta hai. $- 2 (Σ_{r} Σ_{g})^{1/2}$ term ek geometric mean hai jo covariance structure differences ko penalize karta hai.

Worked Example

Worked example 2D Features ke saath Intuition

Imagine karo ki Inception features sirf 2D hain (visualization ke liye). Hamare paas hai: Real images: $μ_{r} = [5, 5]$ par centered, spread $Σ_{r} = [2002]$ (circular)

Generated images: $μ_{g} = [6, 4]$ par centered, spread $Σ_{g} = [4001]$ (elliptical, zyada horizontal variance)

Step 1: Mean difference $∣ μ_{r} - μ_{g} ∣ ∣^{2} = ∣∣ [5, 5] - [6, 4] ∣ ∣^{2} = (- 1)^{2} + (1)^{2} = 2$

Kyun? Generated distribution off-center shift hui hai.

Step 2: Covariance term $Σ_{r} Σ_{g} = [2002] [4001] = [8002]$

$(Σ_{r} Σ_{g})^{1/2} = [8002]$

$Tr (Σ_{r} + Σ_{g} - 2 (Σ_{r} Σ_{g})^{1/2}) = (2 + 4 + 2 + 1) - 2 (8 + 2)$ $= 9 - 2 (2.83 + 1.41) = 9 - 8.48 = 0.52$

Step 3: Total FID $FID = 2 + 0.52 = 2.52$

Yeh kyun matter karta hai: Mean shift sabse zyada contribute karta hai (2), lekin shape mismatch (0.52) bhi matter karta hai. Agar generated images ki covariance structure same hoti, toh FID lower hota.

IS vs FID: Kab Kya Use Karein

| Metric | Kya Measure Karta Hai | Pros | Cons | Best For | |--------|-------|------|----------| | IS | Quality & diversity (bina reference ke) | Fast, real data ki zaroorat nahi | Mode collapse detect nahi karta, ImageNet classes ki taraf biased | Quick sanity check, unconditional generation | | FID | Real data se distribution match | Mode collapse pakad leta hai, human judgment se align hota hai | Real data chahiye, compute karna expensive | Final evaluation, models compare karna |

Connections

GAN Training Dynamics: Yeh metrics GAN training guide karte hain—collapse FID drop mein dikha deta hai.
Inception-v3 Architecture: Feature extractor ko samajhna clarify karta hai ki yeh metrics kyun kaam karte hain.
Mode Collapse in GANs: FID is failure mode detect karne ka primary tool hai.
Precision and Recall for Generative Models: Newer metrics jo FID ko quality (precision) aur coverage (recall) mein decompose karte hain.
Perceptual Loss Functions: Similar idea—pre-trained networks use karke perceptual similarity measure karo.

Recall Ek 12-Saal ke Bachche ko Explain Karo

Imagine karo tum ek teacher ho jo art students grade kar rahe ho. Tum chahte ho ki unki drawings:

Achhi hon (real cheezein jaisi lagein)
Diverse hon (sab ek hi tree na banayein) Lekin tum khud hazaron drawings nahi dekh sakte, toh tum ek robot judge (Inception network) train karte ho jo seekh gaya hai ki "achha art" kaisa hota hai.

Inception Score (IS): Tum robot ko ek drawing dikhao. Agar woh kehta hai "yeh definitely ek dog hai!" (confident), toh drawing clear hai. Agar woh bahut saari drawings mein alag-alag cheezein dekhta hai (dogs, cats, cars), toh woh diverse hai. High IS = confident + diverse.

FID: Tumhare paas real photos aur student drawings hain. Tum robot se dono sets describe karwao (bina "dog" ya "cat" kahe, sirf internal descriptions jaise "pointy," "fluffy"). Phir tum measure karo: dono description clouds kitne similar hain? Chhota distance = drawings real photos jaisi lagti hain.

Dono kyun? IS sirf clarity check karta hai. FID check karta hai ki students sahi cheezein bana rahe hain (ek perfect dragon ki drawing "cats ki photos" se match nahi karti).

#flashcards/ai-ml

Inception Score (IS) kaunsi problem solve karta hai? :: Generative models ka automatic evaluation bina manual inspection ke, generated samples ki quality (sharpness) aur diversity dono measure karna ek pre-trained classifier se.

IS kaunsi do properties check karta hai?

1) Quality: individual samples mein low-entropy p(y|x) honi chahiye (confident predictions), 2) Diversity: marginal p(y) mein high entropy honi chahiye (sabhi classes represented).

Inception Score formula likho :: IS = exp(E_x[D_KL(p(y|x) || p(y))]) jahan p(y|x) classifier ki prediction hai generated image x ke liye, aur p(y) average class distribution hai.

IS mein KL divergence ko exponentiate kyun karte hain?

Log-space se ek interpretable scale par convert karne ke liye jahan higher better hai, aur "effective number of classes" wali intuition se match karne ke liye (perfect generation ke liye IS ≈ number of classes).

Inception Score ki main limitation kya hai?

Yeh mode collapse detect nahi kar sakta—ek model jo sirf ek mode ke perfect samples generate karta hai high IS achieve kar sakta hai kyunki p(y|x) phir bhi confident hai, chahe diversity kam ho.

FID kya measure karta hai?

Real image features aur generated image features ke distributions ke beech Inception-v3 ke feature space mein distance, specifically Fréchet distance yeh assume karke ki dono Gaussian hain.

FID formula likho

FID = ||μ_r - μ_g||² + Tr(Σ_r + Σ_g - 2(Σ_r Σ_g)^(1/2)) jahan μ aur Σ real (r) aur generated (g) features ka mean aur covariance hain.

FID ke do components kya hain?

1) Mean difference ||μ_r - μ_g||² jo distribution shift measure karta hai, 2) Covariance term Tr(Σ_r + Σ_g - 2(Σ_r Σ_g)^(1/2)) jo shape/spread mismatch measure karta hai.

Raw pixels ki jagah Inception-v3 features kyun use karte hain?

Inception features semantic properties (textures, object parts) capture karti hain jo human perception se align hoti hain, unlike pixel-space jahan chhote shifts bade distances cause karte hain chahe perceptually similar ho.

Kya aap alag-alag datasets ke FID scores compare kar sakte hain?

Nahi, FID dataset-dependent hai kyunki intrinsic complexity vary karti hai (MNIST vs CelebA) aur Inception ka feature space kuch data types ke liye doosron se better suited ho sakta hai.

FID = 0 ka matlab kya hai?

Generated aur real distributions feature space mein identical hain—lekin iska matlab yeh ho sakta hai ki model ne training data memorize kar li (overfitting) true generalization nahi.

FID mode collapse detect karne mein IS se better kyun hai?

FID real data distribution se compare karta hai, toh agar generator real data mein maujood modes miss karta hai, toh Σ_g, Σ_r se match nahi karega aur FID high hoga.

FID ke liye Inception-v3 ki kaunsi layer use hoti hai?

Pool3 layer (2048-dimensional feature vectors) final classification layer se pehle, jo high-level semantic features capture karti hai.

FID compute karne ke liye typically kitne samples use hote hain?

Stable covariance estimates paane ke liye real aur generated distributions dono se kam se kam 10,000 samples.

IS specifically KL divergence kyun use karta hai?

KL divergence naturally conditional p(y|x) aur marginal p(y) ke beech relationship capture karta hai—yeh bada hota hai jab individual samples confident hoon (low H(y|x)) lekin overall distribution diverse ho (high H(y)).

FID mein trace operation kya hai?

Tr(A) matrix A ke diagonal elements ko sum karta hai, is context mein sabhi 2048 feature dimensions mein total variance represent karta hai.

FID mein (Σ_r Σ_g)^(1/2) ki zaroorat kyun hai?

Matrix square root covariances ka geometric mean provide karta hai, ek symmetric distance create karta hai jo distributions ke beech correlation structure differences ko properly account karta hai.

High IS ke saath high FID kya indicate karta hai?

Generated images individually sharp hain aur bahut saari classes cover karti hain (high IS) lekin real data distribution se match nahi karti (high FID)—possible hai agar plausible lekin galat samples generate ho raha ho.

Low IS ke saath low FID kya indicate karta hai?

Yeh contradictory aur unlikely hai—low IS matlab blurry ya mode-collapsed samples, jo real data se achhi tarah match nahi hone chahiye. Evaluation error suggest karta hai.

FID ko quality se diversity alag karne ke liye kaunsa metric complement karta hai?

Generative models ke liye Precision aur Recall—precision sample quality measure karta hai (no outliers), recall mode coverage measure karta hai (distribution support).