Evaluating generative models (FID, IS)

Overview

Evaluating generative models is fundamentally different from discriminative tasks because there's no single "correct" output—we care about quality (realism) and diversity (coverage of the data distribution). Traditional metrics like accuracy don't apply, so we use specialized scores: Inception Score (IS) and Fréchet Inception Distance (FID).

Inception Score (IS)

The Core Idea

Inception Score measures quality and diversity by checking:

1. Sharpness (quality): Each generated image should be confidently classified into one category → low entropy conditional distribution $p(y|x)$
2. Diversity: The overall label distribution should be uniform → high entropy marginal $p(y)$

Derivation from First Principles

Step 1: What makes a "good" generative model?

A good generator should produce:

• Meaningful images: If you show one image to a classifier, it should be confident (e.g., "90% dog, 10% other") → $p(y|x)$ has low entropy
• Diverse images: Across all images, you should see all classes → $p(y)$ has high entropy

Step 2: KL divergence quantifies this

$D_{\text{KL}}(p(y|x) \, || \, p(y)) = \sum_y p(y|x) \log \frac{p(y|x)}{p(y)}$

Why this step? KL divergence is large when:

• $p(y|x)$ is peaked (low entropy) → good quality
• $p(y)$ is uniform (high entropy) → good diversity

Step 3: Average over all generated samples

$\text{KL}_{\text{avg}} = \mathbb{E}_{x \sim p_g} \left[ \sum_y p(y|x) \log \frac{p(y|x)}{p(y)} \right]$

We split this using logarithm properties:

$= \mathbb{E}_{x \sim p_g} \left[ \sum_y p(y|x) \log p(y|x) \right] - \mathbb{E}_{x \sim p_g} \left[ \sum_y p(y|x) \log p(y) \right]$

The first term is $-H(y|x)$ (negative conditional entropy), the second simplifies because $\mathbb{E}_x[\sum_y p(y|x) \log p(y)] = \sum_y p(y) \log p(y) = -H(y)$.

So: $\text{KL}_{\text{avg}} = H(y) - H(y|x)$

Higher $H(y)$ (diverse labels) and lower $H(y|x)$ (confident predictions) → higher IS.

Step 4: Exponentiate for interpretability

$\text{IS} = \exp(\text{KL}_{\text{avg}})$

Why exponentiate? To convert from log-space to a scale where higher = better, and to make it match "effective number of classes" intuition. IS≈ 1 means terrible (all images look the same or are noise), IS ≈ number of classes means perfect.

Worked Example

Fréchet Inception Distance (FID)

The Core Idea

FID measures the distance between the distribution of real images and generated images in the feature space of Inception-v3. Lower FID = distributions are closer = better generation.

Derivation from First Principles

Step 1: The Fréchet distance for Gaussians

If two distributions are Gaussian: $p_r \sim \mathcal{N}(\mu_r, \Sigma_r), \quad p_g \sim \mathcal{N}(\mu_g, \Sigma_g)$

The Fréchet distance (also called Wasserstein-2 distance for Gaussians) has a closed form:

$d_F^2(\mathcal{N}(\mu_1, \Sigma_1), \mathcal{N}(\mu_2, \Sigma_2)) = |\mu_1 - \mu_2||^2 + \text{Tr}(\Sigma_1 + \Sigma_2 - 2(\Sigma_1 \Sigma_2)^{1/2})$

Why this formula? It comes from the 2-Wasserstein distance:

$W_2(p, q) = \inf_{\pi} \mathbb{E}_{(x,y) \sim \pi}[||x - y||^2]$

where $\pi$ is a coupling between distributions. For Gaussians, this infimum has analytical solution.

Step 2: Apply to feature distributions

Extract features:

• For $N$ real images: $\{f_r^{(i)}\}_{i=1}^N$ from Inception pool3 layer (2048-dim)
• For $N$ fake images: $\{f_g^{(i)}\}_{i=1}^N$

Estimate parameters: $\mu_r = \frac{1}{N}\sum_i f_r^{(i)}, \quad \Sigma_r = \frac{1}{N-1}\sum_i (f_r^{(i)} - \mu_r)(f_r^{(i)} - \mu_r)^T$

(Same for $\mu_g, \Sigma_g$)

Step 3: Compute matrix square root

The term $(\Sigma_r \Sigma_g)^{1/2}$ is computed via eigenvalue decomposition:

If $\Sigma_r \Sigma_g = U \Lambda U^T$, then $(\Sigma_r \Sigma_g)^{1/2} = U \Lambda^{1/2} U^T$

Why? Because for positive semi-definite matrices, the square root is well-defined.

Step 4: Trace of the difference

$\text{Tr}(\Sigma_r + \Sigma_g - 2(\Sigma_r \Sigma_g)^{1/2})$

Why trace? It sums the covariance mismatch across all 2048 dimensions. The $-2(\Sigma_r \Sigma_g)^{1/2}$ term is a geometric mean that penalizes covariance structure differences.

Worked Example

IS vs FID: When to Use What

| Metric | Measures | Pros | Cons | Best For | |--------|-------|------|----------| | IS | Quality & diversity (without reference) | Fast, no real data needed | Doesn't detect mode collapse, biased toward ImageNet classes | Quick sanity check, unconditional generation | | FID | Distribution match to real data | Catches mode collapse, aligns with human judgment | Requires real data, expensive to compute | Final evaluation, comparing models |

Connections

• GAN Training Dynamics: These metrics guide GAN training—collapse shows in FID drop.
• Inception-v3 Architecture: Understanding the feature extractor clarifies why these metrics work.
• Mode Collapse in GANs: FID is the primary tool for detecting this failure mode.
• Precision and Recall for Generative Models: Newer metrics that decompose FID into quality (precision) and coverage (recall).
• Perceptual Loss Functions: Similar idea—use pre-trained networks to measure perceptual similarity.

#flashcards/ai-ml

What problem does Inception Score (IS) solve? :: Automatic evaluation of generative models without manual inspection, measuring both quality (sharpness) and diversity of generated samples using a pre-trained classifier.

Write the Inception Score formula :: IS = exp(E_x[D_KL(p(y|x) || p(y))]) where p(y|x) is the classifier's prediction for generated image x, and p(y) is the average class distribution.

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)