Evaluating generative models (FID, IS): Exercises

These problems climb a ladder: from recognising what each symbol means, to applying the formulas by hand, to analysing their failure modes, to synthesising both scores into a judgement, and finally mastering the edge cases. Do each one on paper first, then open the solution.

Everything here builds on Evaluating generative models (FID, IS). If a symbol looks unfamiliar, that parent note defines it — but every solution below re-earns the tools it uses.

Figure — Evaluating generative models (FID, IS)

Level 1 — Recognition

Exercise 1.1

For each quantity, say whether higher is better or lower is better, and in one phrase what it measures: (a) Inception Score, (b) Fréchet Inception Distance, (c) the conditional entropy $H (y ∣ x)$ , (d) the marginal entropy $H (y)$ .

Recall Solution

Recall the two building blocks. $H (y ∣ x)$ is the uncertainty of the classifier about one image — small means "the classifier is confident". $H (y)$ is the uncertainty about which class you'll see across all images — large means "many different classes appear". And $IS = exp (H (y) - H (y ∣ x))$ .

(a) IS: higher is better — measures quality × diversity.
(b) FID: lower is better — measures distance between real and fake feature-clouds.
(c) $H (y ∣ x)$ : lower is better — confident, sharp images.
(d) $H (y)$ : higher is better — diverse class coverage.

Exercise 1.2

In the FID formula $FID = ∥ μ_{r} - μ_{g} ∥^{2} + Tr (Σ_{r} + Σ_{g} - 2 (Σ_{r} Σ_{g})^{1/2}),$ name what each piece is and which mismatch it detects.

Recall Solution

$μ_{r}, μ_{g}$ are the mean feature vectors (the centre of each cloud). $∥ μ_{r} - μ_{g} ∥^{2}$ is the squared distance between centres — it fires when the fake images live in the wrong region of feature space (wrong "average look").
$Σ_{r}, Σ_{g}$ are the covariance matrices — how the clouds are stretched and correlated. $Tr (\dots)$ sums the shape/spread mismatch across all feature dimensions.
$(Σ_{r} Σ_{g})^{1/2}$ is a matrix geometric mean; the $- 2 (\dots)$ term shrinks the trace toward zero exactly when the two shapes agree.

Level 2 — Application

Exercise 2.1

A generator makes 2 images over a 2-class classifier: $p (y ∣ x_{1}) = [0.8, 0.2], p (y ∣ x_{2}) = [0.2, 0.8] .$ Compute the Inception Score. Use natural logarithms.

Recall Solution

Step 1 — marginal $p (y)$ (WHAT: the average label distribution; WHY: IS rewards this being uniform). $p (y) = \frac{1}{2} ([0.8, 0.2] + [0.2, 0.8]) = [0.5, 0.5] .$

Step 2 — KL divergence per image. Recall $D_{KL} (p ∥ q) = \sum_{y} p (y) ln \frac{p ( y )}{q ( y )}$ . For image 1: $0.8 ln \frac{0.8}{0.5} + 0.2 ln \frac{0.2}{0.5} = 0.8 (0.4700) + 0.2 (- 0.9163) = 0.3760 - 0.1833 = 0.1927.$ By symmetry image 2 gives the same $0.1927$ .

Step 3 — average and exponentiate. $IS = exp (\frac{1}{2} (0.1927 + 0.1927)) = exp (0.1927) \approx 1.213 .$ Interpretation: only mildly better than the worst case ( $IS = 1$ ) because each image is only somewhat confident (0.8, not 0.99).

Exercise 2.2

Two 1-D feature clouds (so covariance is just a variance): $μ_{r} = 3, σ_{r}^{2} = 4, μ_{g} = 5, σ_{g}^{2} = 1.$ Compute FID. In 1-D, $(Σ_{r} Σ_{g})^{1/2} = σ_{r}^{2} σ_{g}^{2} = σ_{r} σ_{g}$ .

Recall Solution

Step 1 — mean term (WHAT: distance between centres). $∥ μ_{r} - μ_{g} ∥^{2} = (3 - 5)^{2} = 4.$

Step 2 — covariance term. In 1-D the trace is just the number itself: $σ_{r}^{2} + σ_{g}^{2} - 2 σ_{r}^{2} σ_{g}^{2} = 4 + 1 - 2 4 \cdot 1 = 5 - 2 (2) = 1.$ Notice this is $(σ_{r} - σ_{g})^{2} = (2 - 1)^{2} = 1$ — a perfect square, always $\geq 0$ .

Step 3 — sum. $FID = 4 + 1 = 5 .$

Level 3 — Analysis

Exercise 3.1 — Mode collapse vs IS

A GAN suffers total mode collapse: it generates the same one perfect, razor-sharp dog every time. Over a 3-class classifier every sample gives $p (y ∣ x) = [1, 0, 0]$ . (a) Compute $H (y)$ , $H (y ∣ x)$ , and IS. (b) Explain why IS fails here, and which score catches it.

Recall Solution

(a) With every image identical:

$p (y) = [1, 0, 0]$ , so $H (y) = - \sum p ln p = - 1 ln 1 = 0$ (using $0 ln 0 = 0$ ).
Each $p (y ∣ x) = [1, 0, 0]$ , so $H (y ∣ x) = 0$ too.
$IS = exp (H (y) - H (y ∣ x)) = exp (0 - 0) = exp (0) = 1$ .

Here IS does bottom out at 1 because collapse to a single class kills diversity ( $H (y) = 0$ ). But watch the more dangerous case below.

(b) The truly deceptive collapse is within-class diversity loss: a GAN produces confident images spread across all classes but only one image per class (like the parent note's toy example, IS ≈ 2 — "reasonable"). IS sees diverse labels and rewards it, blind to the fact that the model only knows 3 pictures. FID catches this: the real dog cloud is broad (many dog poses), the fake cloud is a single point, so $Σ_{g}$ is nearly zero, the shape mismatch term blows up, and FID is large. See Mode Collapse in GANs and Precision and Recall for Generative Models.

Exercise 3.2 — Non-negativity proof

Prove the covariance term $σ_{r}^{2} + σ_{g}^{2} - 2 σ_{r} σ_{g} \geq 0$ in the 1-D FID, and state when it equals zero.

Recall Solution

WHAT/WHY: we want to show FID never rewards a shape mismatch with a negative number. $σ_{r}^{2} + σ_{g}^{2} - 2 σ_{r} σ_{g} = (σ_{r} - σ_{g})^{2} \geq 0,$ since any real square is non-negative. It equals 0 iff $σ_{r} = σ_{g}$ — the spreads match exactly. Combined with $∥ μ_{r} - μ_{g} ∥^{2} \geq 0$ (equal iff $μ_{r} = μ_{g}$ ), FID $= 0$ iff the Gaussians are identical, which is exactly the property a distance must have.

Level 4 — Synthesis

Exercise 4.1 — Reading two models together

	IS	FID
Model A	9.2	40
Model B	8.0	12
Which model do you ship, and why? What single dataset fact would flip your answer?

Recall Solution

Step 1 — trust FID for the distribution match. FID compares to real data, so B's 12 ≪ 40 says B's feature cloud sits much closer to the real one: better realism and better coverage. A's higher IS only means A's images are more confidently classifiable — which rewards sharpness even if unrealistic. Step 2 — ship B. Lower FID almost always tracks better perceived quality; A's IS edge is the classic "sharp-but-off-distribution" trap. Step 3 — what flips it: if the target dataset is narrow / single-domain (e.g. only faces) so that Inception-v3's ImageNet feature space is poorly matched, FID becomes noisy and unreliable; you'd then weight human eval / precision-recall more, and A could win if its samples are visibly better. Also: if you require class-diversity and B is secretly collapsed within-class, IS + Precision and Recall for Generative Models would expose it.

Exercise 4.2 — Sample-size bias

FID is estimated from finite samples. If you compute FID with $N = 1, 000$ images instead of $N = 50, 000$ , does the estimate tend to be higher or lower than the true FID? Explain the mechanism.

Recall Solution

Higher (positively biased). WHY: with few samples the estimated covariances $\hat{Σ}_{r}, \hat{Σ}_{g}$ are noisy, and the estimated means wander from the truth. Both the mean term $∥ \overset{μ}{^}_{r} - \overset{μ}{^}_{g} ∥^{2}$ and the shape term pick up extra apparent mismatch from sampling noise, and since FID $\geq 0$ this noise can only push the score up, never below the true value. Consequence: FID numbers are only comparable at a fixed, large $N$ (that is why the field standardises on 50k). This is an estimation bias, not a property of the true Fréchet distance.

Level 5 — Mastery

Exercise 5.1 — Full 2-D FID by hand

Real cloud: $μ_{r} = [0, 0]$ , $Σ_{r} = [4001]$ . Generated cloud: $μ_{g} = [1, 2]$ , $Σ_{g} = [1001]$ . Compute FID exactly.

Recall Solution

Step 1 — mean term. $∥ μ_{r} - μ_{g} ∥^{2} = (0 - 1)^{2} + (0 - 2)^{2} = 1 + 4 = 5.$

Step 2 — matrix product then square root (WHY: covariance term needs $(Σ_{r} Σ_{g})^{1/2}$ ). Both matrices are diagonal, so multiply entrywise: $Σ_{r} Σ_{g} = [4 \cdot 1 0 0 1 \cdot 1] = [4001] .$ For a diagonal PSD matrix the square root is the entrywise square root: $(Σ_{r} Σ_{g})^{1/2} = [2001] .$

Step 3 — trace term. $Σ_{r} + Σ_{g} - 2 (Σ_{r} Σ_{g})^{1/2} = [4 + 1 - 2 \cdot 2 0 0 1 + 1 - 2 \cdot 1] = [1000] .$ $Tr = 1 + 0 = 1.$

Step 4 — sum. $FID = 5 + 1 = 6 .$

Exercise 5.2 — Degenerate covariance (edge case)

Now let the generator collapse in one direction: $Σ_{g} = [1000]$ (zero variance along axis 2), keeping $μ_{r} = μ_{g} = [0, 0]$ and $Σ_{r} = [4001]$ . Compute FID and interpret the collapse.

Recall Solution

Means match, so the first term is $0$ . $Σ_{r} Σ_{g} = [4000], (Σ_{r} Σ_{g})^{1/2} = [2000] .$ $Σ_{r} + Σ_{g} - 2 (\dots) = [4 + 1 - 4 0 0 1 + 0 - 0] = [1001], Tr = 2.$ $FID = 0 + 2 = 2 .$ Interpretation: even with matching centres, the vanished spread along axis 2 (variance $0$ vs the real $1$ ) contributes exactly $(\sqrt1 - \sqrt0)^{2} = 1$ , and the mismatched spread on axis 1 contributes another $1$ . FID correctly penalises the collapsed direction — the geometric picture of within-mode collapse that IS cannot see. Compare this feature-space reasoning to Perceptual Loss Functions and Inception-v3 Architecture.

Recall Quick self-check

Which score never looks at real data? ::: Inception Score (IS) FID is zero exactly when? ::: the two Gaussians are identical ( $μ_{r} = μ_{g}$ and $Σ_{r} = Σ_{g}$ ) A within-class-collapsed generator: high or low IS? ::: can be high (deceptive) — FID catches it Finite-sample FID is biased which way? ::: upward (higher than the true value)