Evaluating generative models (FID, IS): Worked examples

You have already met the two headline scores on the parent note Evaluating generative models (FID, IS):

IS (Inception Score) — bigger is better, measures sharpness + spread of labels.
FID (Fréchet Inception Distance) — smaller is better, measures distance between feature clouds.

This page does one thing: it hunts down every case class these scores can throw at you and works each one to the last decimal. No new theory — just relentless coverage. If you have not seen $lo g$ , $exp$ , or a matrix before, each is re-earned the moment it appears.

Before anything else, one symbol we will lean on constantly:

The scenario matrix

Think of "cases" as the different shapes of input a metric can receive. If we only ever practise the nice symmetric case, we get ambushed by the degenerate ones. Here is the full board we will clear:

Cell	Metric	Scenario class	What could surprise you
A	IS	Perfect, balanced generator	The "textbook" answer — a reference point
B	IS	Mode collapse (all one class)	IS stays high — the famous blind spot
C	IS	Pure noise / uniform predictions	IS crashes to $1$ (the floor)
D	IS	Confident but imbalanced labels	Diversity term drags the score down
E	FID	Identical distributions	FID $= 0$ (the floor)
F	FID	Mean shift only, same spread	Only the $∥ μ_{r} - μ_{g} ∥^{2}$ term fires
G	FID	Same mean, different spread	Only the trace term fires
H	FID	Degenerate: zero variance ( $Σ = 0$ )	Square-root term vanishes
I	Word problem	Two GAN checkpoints, pick the winner	IS and FID disagree
J	Exam twist	Add a duplicated sample	Which metric notices?

We now clear the board cell by cell.

IS cases (cells A–D)

Everywhere below, $N$ is the batch size — the number of generated images we score together (in practice $N = 50, 000$ ; in our toy cells $N = 3$ or $4$ ). $p (y ∣ x)$ is the classifier's probability vector for one image — "how sure am I this is class $y$ ?" — and

$p (y) = \frac{1}{N} \sum_{i = 1}^{N} p (y ∣ x_{i})$

is the marginal, the average of those $N$ vectors: "across the whole batch, how often does each class show up?" The score is

$IS = exp (\frac{1}{N} \sum_{i = 1}^{N} \sum_{y} p (y ∣ x_{i}) lo g \frac{p ( y ∣ x _{i} )}{p ( y )}) .$

We use $lo g$ (natural log, base $e$ ) throughout so that $exp$ cleanly undoes it — that is the only reason the base- $e$ pairing is chosen: $exp (lo g v) = v$ , no leftover constants.

Cell A — the balanced, confident generator

Worked example A · Three sharp images, one per class

Predictions over 3 classes ( $K = 3$ , $N = 3$ ):

$p (y ∣ x_{1}) = [0.8, 0.1, 0.1]$
$p (y ∣ x_{2}) = [0.1, 0.8, 0.1]$
$p (y ∣ x_{3}) = [0.1, 0.1, 0.8]$

Forecast: each image is confident and each class appears once — so we expect an IS comfortably above $1$ but below the ceiling of $3$ . Guess before reading.

Step 1 — marginal $p (y)$ . Average the three vectors. $p (y) = \frac{1}{3} ([0.8, 0.1, 0.1] + [0.1, 0.8, 0.1] + [0.1, 0.1, 0.8]) = [0.333, 0.333, 0.333] .$ Why this step? The KL needs something to compare each image against. Uniform marginal = maximal diversity, the ideal denominator.

Step 2 — KL for image 1. $D_{KL} = \sum_{y} p (y ∣ x) lo g \frac{p ( y ∣ x )}{p ( y )}$ . $0.8 lo g \frac{0.8}{0.333} + 0.1 lo g \frac{0.1}{0.333} + 0.1 lo g \frac{0.1}{0.333} .$ The two small terms are identical, so we write them as $2 \times 0.1 \times lo g \frac{0.1}{0.333} = 2 \times 0.1 \times (- 1.204) = - 0.241$ . The big term is $0.8 \times 0.875 = 0.700$ . Total: $0.700 - 0.241 = 0.459.$ Why this step? KL rewards a peaked $p (y ∣ x)$ sitting on top of a flat $p (y)$ . That gap is exactly "sharp image + diverse batch".

Step 3 — symmetry + exponentiate. All three images are permutations, so all give $0.459$ . Average $= 0.459$ . $IS = exp (0.459) \approx 1.58.$

Verify: $1 < 1.58 < 3$ ✓ — above the noise floor, below the perfect ceiling of "number of classes" ( $K = 3$ ). Matches forecast.

Worked example B · All three images are the SAME class

Predictions:

$p (y ∣ x_{1}) = p (y ∣ x_{2}) = p (y ∣ x_{3}) = [0.9, 0.05, 0.05]$

Forecast: a broken generator producing only dogs. Surely IS punishes this? Guess "IS near 1"… then watch.

Step 1 — marginal. Averaging three identical vectors just returns the vector: $p (y) = [0.9, 0.05, 0.05] .$ Why this step? This is the crux: when every image agrees, the marginal equals the conditional.

Step 2 — KL. With $p (y ∣ x) = p (y)$ , every ratio $\frac{p ( y ∣ x )}{p ( y )} = 1$ , and $lo g 1 = 0$ : $D_{KL} = 0.9 lo g 1 + 0.05 lo g 1 + 0.05 lo g 1 = 0.$ Why this step? When conditional and marginal coincide there is no information gained by seeing the image — KL is defined to be zero exactly then.

Step 3. $IS = exp (0) = 1$ .

Verify: IS $= 1$ , the floor. Here IS actually catches collapse — because our toy batch is tiny and perfectly identical.

The sidebar warning is that this is too clean. Real collapse never produces byte-identical vectors — the classifier wobbles slightly per image. Let us see concretely how that wobble rescues IS from the floor, wrongly.

Worked example B2 · Realistic collapse with tiny per-image jitter

Same "all dogs" generator, but now the 3-class predictions wobble a little:

$p (y ∣ x_{1}) = [0.90, 0.06, 0.04]$
$p (y ∣ x_{2}) = [0.88, 0.05, 0.07]$
$p (y ∣ x_{3}) = [0.92, 0.04, 0.04]$

Forecast: still all dogs — diversity is dead. Will IS stay at the floor of $1$ ? Guess.

Step 1 — marginal. Average componentwise: $p (y) = \frac{1}{3} ([0.90, 0.06, 0.04] + [0.88, 0.05, 0.07] + [0.92, 0.04, 0.04]) = [0.9000, 0.0500, 0.0500] .$ Why this step? Because the vectors are not identical, the marginal no longer equals any single conditional — the $lo g 1 = 0$ trick from Cell B breaks.

Step 2 — KL per image (image 1). $0.90 lo g \frac{0.90}{0.90} + 0.06 lo g \frac{0.06}{0.05} + 0.04 lo g \frac{0.04}{0.05} = 0 + 0.06 (0.1823) + 0.04 (- 0.2231) = 0.01094 - 0.00892 = 0.00202.$ The other two images give similarly tiny positive KLs. Why this step? Any mismatch between conditional and marginal produces a small positive KL — no longer exactly zero.

Step 3 — average + exponentiate. The three KLs average to about $0.00350$ , so $IS = exp (0.00350) \approx 1.0035.$

Verify: IS $\approx 1.0035 > 1$ ✓. In our tiny batch the lift is small, but on 50k jittery samples these micro-mismatches accumulate and IS can climb into the healthy single or double digits while diversity is genuinely dead. That is precisely why we never trust IS alone — see Mode Collapse in GANs and pair with Precision and Recall for Generative Models.

Cell C — pure noise (uniform predictions)

Cell D — confident but imbalanced

Worked example D · Sharp images, but two-thirds are dogs

Predictions ( $K = 2$ classes, "dog" vs "cat"):

$p (y ∣ x_{1}) = [0.95, 0.05]$
$p (y ∣ x_{2}) = [0.95, 0.05]$
$p (y ∣ x_{3}) = [0.05, 0.95]$

Forecast: each image is very confident (quality high) but the batch leans dog (diversity mediocre). Expect an IS above $1$ but dragged down versus a balanced version.

Step 1 — marginal. $p (y) = \frac{1}{3} ([0.95, 0.05] + [0.95, 0.05] + [0.05, 0.95]) = [0.65, 0.35] .$ Why this step? The tilt (0.65 vs 0.35) is exactly the diversity penalty — a flatter marginal would score higher.

Step 2 — KL per image. Dog images ( $x_{1}, x_{2}$ ): $0.95 lo g \frac{0.95}{0.65} + 0.05 lo g \frac{0.05}{0.35} = 0.95 (0.3795) + 0.05 (- 1.9459) = 0.3605 - 0.0973 = 0.2632.$ Cat image ( $x_{3}$ ): $0.05 lo g \frac{0.05}{0.65} + 0.95 lo g \frac{0.95}{0.35} = 0.05 (- 2.5649) + 0.95 (0.9985) = - 0.1282 + 0.9486 = 0.8203.$ Why this step? We compute each image separately because the exponent averages per-image KLs; the dog images share the score, the lone cat scores higher against the dog-heavy marginal.

Step 3 — average + exponentiate. $\overline{KL} = \frac{0.2632 + 0.2632 + 0.8203}{3} = 0.4489, IS = exp (0.4489) \approx 1.567.$

Verify: A balanced 2-class version ( $[0.95, 0.05]$ and $[0.05, 0.95]$ each once) gives marginal $[0.5, 0.5]$ and IS $\approx e^{0.5570} \approx 1.745$ . Our imbalanced batch scores lower ( $1.567$ ) ✓ — the diversity term did its job.

FID cases (cells E–H)

Now switch metric. We drop into feature space — the 2048 numbers Inception-v3's pool3 layer emits per image (see Inception-v3 Architecture and why perceptual features beat raw pixels, Perceptual Loss Functions). For hand-work we pretend features are 2-dimensional so we can draw them.

The figure below shows the whole idea at a glance: three toy feature clouds. The magenta cloud is the real data centred at $μ_{r} = (5, 5)$ . The violet cloud is a generator whose samples have the same spread but a shifted centre $μ_{g} = (6, 4)$ — the little navy arrow between the two X markers is the "mean gap" that FID's first term measures (this is Cell F). The orange cloud sits somewhere else and is visibly tighter — a low-diversity generator whose small spread is what FID's trace term catches (the flavour of Cells G and H). Keep this picture in mind: FID literally adds "how far apart are the centres" to "how differently do they spread".

Figure — Evaluating generative models (FID, IS)

We model each cloud as a Gaussian with mean $μ$ (its centre) and covariance $Σ$ (its spread + tilt). The score:

$FID = centre gap ∥ μ_{r} - μ_{g} ∥^{2} + spread mismatch Tr (Σ_{r} + Σ_{g} - 2 (Σ_{r} Σ_{g})^{1/2}) .$

$Tr$ ("trace") just means "add up the diagonal entries of a matrix" — for a $2 \times 2$ matrix, top-left plus bottom-right. We use it because the diagonal sums the mismatch across every feature dimension at once.

Definition The matrix square root

$(Σ_{r} Σ_{g})^{1/2}$ For an ordinary number, "square root of $9$ " is the number that, multiplied by itself, gives $9$ (namely $3$ ). A matrix square root of a matrix $M$ is the same idea: a matrix $S$ with $S S = M$ . It exists and is unique when $M$ is positive semi-definite (roughly, a matrix that never "flips" a vector's direction against itself — covariance products qualify). How you actually compute it (general case): you diagonalise. Write $M = U Λ U^{T}$ , where $Λ$ is a diagonal matrix of eigenvalues $λ_{i}$ and $U$ holds the eigenvectors. Then take the ordinary square root of each eigenvalue on the diagonal: $M^{1/2} = U Λ^{1/2} U^{T}, Λ^{1/2} = diag (λ_{1}, λ_{2}, \dots) .$ Intuition: the eigenvectors are the "natural axes" of the spread; along each axis the matrix just stretches by $λ_{i}$ , so its square root stretches by $λ_{i}$ . In our toy cells the matrices are already diagonal, so $U$ is the identity and this collapses to "square-root each diagonal entry" — but the machinery above is what runs when the covariance is tilted (off-diagonal entries non-zero), which is the real 2048-dim case. In code: scipy.linalg.sqrtm.

Cell E — identical clouds (the floor)

Worked example E · Real and fake are the same distribution

$μ_{r} = μ_{g} = [5, 5]$ , $Σ_{r} = Σ_{g} = [2002]$ .

Forecast: if the generator perfectly matches, FID must be its floor. Which value?

Step 1 — centre gap. $∥[5, 5] - [5, 5] ∥^{2} = 0$ . Why this step? No shift means the first term contributes nothing; we isolate the spread term.

Step 2 — spread term. Here $Σ_{r} Σ_{g} = [4004]$ (diagonal, so eigenvalues are $4$ and $4$ ); each eigenvalue's square root is $2$ , giving $(Σ_{r} Σ_{g})^{1/2} = [2002]$ . Then $Σ_{r} + Σ_{g} - 2 (Σ_{r} Σ_{g})^{1/2} = [2002] + [2002] - 2 [2002] = [0000] .$ Why the matrix square root? $(Σ_{r} Σ_{g})^{1/2}$ is the geometric mean of the two spreads; when they're equal it lands exactly between them and the whole term cancels.

Step 3. $Tr (0) = 0$ , so $FID = 0 + 0 = 0$ .

Verify: FID $= 0$ — the floor. A perfect match scores zero, confirming "lower is better." ✓

Cell F — mean shift only

Worked example F · Same spread, shifted centre

$μ_{r} = [5, 5]$ , $μ_{g} = [6, 4]$ , and $Σ_{r} = Σ_{g} = [2002]$ .

Forecast: clouds are the same shape but sit in different spots. Only the centre-gap term should fire. Predict its value.

Step 1 — centre gap. $[5, 5] - [6, 4] = [- 1, 1]$ , squared length $= (- 1)^{2} + 1^{2} = 2$ . Why this step? This is the magenta→violet arrow in the figure; a pure translation lives entirely in the first term.

Step 2 — spread term. Covariances identical → same cancellation as Cell E → trace $= 0$ . Why this step? Identical spreads means the geometric-mean term equals both, so the mismatch is zero — nothing to add.

Step 3. $FID = 2 + 0 = 2$ .

Verify: FID $= 2$ . Distance is squared Euclidean, so it has units of feature² — a pure translation of the cloud is caught entirely by the first term. ✓ See figure: the magenta cloud vs the violet cloud.

Cell G — spread mismatch only

Worked example G · Same centre, different spread

$μ_{r} = μ_{g} = [0, 0]$ , $Σ_{r} = [4004]$ (wide), $Σ_{g} = [1001]$ (tight).

Forecast: the generator makes low-variety samples clustered at the right centre — classic mild collapse. Centre gap is zero, so the trace term must expose it.

Step 1 — centre gap. $0$ . Why this step? Same centres kill the first term, leaving the spread mismatch fully responsible — this is the "tight orange cloud" idea from the figure.

Step 2 — spread term. $Σ_{r} Σ_{g} = [4004]$ , eigenvalues $4, 4$ , so its square root is $[2002]$ . $Σ_{r} + Σ_{g} - 2 (Σ_{r} Σ_{g})^{1/2} = [4004] + [1001] - 2 [2002] = [1001] .$ Why this step? Because the geometric mean $4 \cdot 1 = 2$ sits between the wide and tight spreads, subtracting twice it leaves a positive residual — that residual is the diversity gap.

Step 3. $Tr [1001] = 1 + 1 = 2$ , so $FID = 0 + 2 = 2$ .

Verify: Per axis this is $4 + 1 - 2 4 \cdot 1 = 5 - 4 = 1$ ; two axes give $2$ ✓. Here FID does catch reduced diversity — exactly where IS was blind in Cell B.

Cell H — degenerate: zero-variance generator

Worked example H · The generator outputs one identical image

$μ_{r} = [0, 0]$ , $Σ_{r} = [2002]$ ; $μ_{g} = [0, 0]$ , $Σ_{g} = [0000]$ (a single point — zero spread).

Forecast: total collapse — zero spread. The square-root term should vanish because anything times the zero matrix is zero.

Step 1 — centre gap. $∥[0, 0] - [0, 0] ∥^{2} = 0$ . Why this step? We placed the point at the real centre on purpose, so only the spread term can react — isolating the degenerate behaviour.

Step 2 — the cross term. $Σ_{r} Σ_{g} = [2002] [0000] = [0000]$ . Its eigenvalues are both $0$ , so the matrix square root is $0$ along each axis, i.e. the zero matrix: $(Σ_{r} Σ_{g})^{1/2} = [0000]$ . Why this step? Multiplying by the zero matrix wipes the geometric-mean cross term, so the penalty reduces to the real cloud's own spread — the metric stays finite even at this edge.

Step 3 — spread term and total. $Σ_{r} + Σ_{g} - 2 (Σ_{r} Σ_{g})^{1/2} = [2002] + [0000] - 2 [0000] = [2002] .$ $Tr [2002] = 2 + 2 = 4$ , so $FID = 0 + 4 = 4$ .

Verify: FID $= 4 = Tr (Σ_{r})$ . A point-generator is penalised by exactly the real cloud's total variance — larger, more diverse real data ⇒ larger penalty. FID stays finite and well-defined even at this degenerate edge. ✓

Cell I — the word problem (IS and FID as tie-breakers)

Worked example I · Which checkpoint ships?

You train two GANs on a 4-class dataset ( $K = 4$ ). Toy batches of 4 images:

Checkpoint P (collapsed to class 1, very sharp): all four images $= [0.97, 0.01, 0.01, 0.01]$ .

Checkpoint Q (balanced, slightly softer): $[0.85, 0.05, 0.05, 0.05]$ , $[0.05, 0.85, 0.05, 0.05]$ , $[0.05, 0.05, 0.85, 0.05]$ , $[0.05, 0.05, 0.05, 0.85]$ .

Real data feature centre (2D toy) $μ_{r} = [0, 0]$ . P's features cluster at $μ_{P} = [3, 0]$ (drifted), Q's at $μ_{Q} = [0, 0]$ ; assume all $Σ = I$ (the $2 \times 2$ identity).

Forecast: guess which checkpoint each metric prefers before computing.

Step 1 — IS for P. Marginal $= [0.97, 0.01, 0.01, 0.01]$ (all identical), so $p (y ∣ x) = p (y)$ , KL $= 0$ , $IS_{P} = exp (0) = 1$ . Why this step? Same collapse mechanism as Cell B — identical images force marginal to equal conditional.

Step 2 — IS for Q. Marginal $= [0.25, 0.25, 0.25, 0.25]$ . Each image's KL: $0.85 lo g \frac{0.85}{0.25} + 3 \cdot 0.05 lo g \frac{0.05}{0.25} = 0.85 (1.2238) + 0.15 (- 1.6094) = 1.0402 - 0.2414 = 0.7988.$ $IS_{Q} = exp (0.7988) \approx 2.223$ . Why this step? A uniform 4-class marginal is the diverse ideal; the sharp-but-spread predictions push IS well above P's floor.

Step 3 — FID for both. $Σ = I$ everywhere, so the trace term is $0$ (identical spreads, Cell E logic). Only centre gaps: $FID_{P} = ∥[0, 0] - [3, 0] ∥^{2} = 9$ ; $FID_{Q} = ∥[0, 0] - [0, 0] ∥^{2} = 0$ . Why this step? With spreads matched, FID reduces to pure centre distance — P has drifted away from the real cloud, Q sits on top of it.

Step 4 — decide. IS: $Q (2.22) > P (1.0)$ . FID: $Q (0) < P (9)$ . Both crown Q, and that agreement is your confidence.

Verify: $IS_{P} = 1$ (floor), $IS_{Q} \approx 2.223$ , $FID_{P} = 9$ , $FID_{Q} = 0$ ✓. The lesson: had you looked at IS alone on a checkpoint whose collapse produced slight per-image jitter (Cell B2), IS could have quietly favoured the collapsed model — FID's mean-and-spread comparison is the tie-breaker.

Cell J — the exam twist (duplicated sample)

Worked example J · You accidentally duplicate one image in the batch

Original 2-class batch ( $K = 2$ , $N = 2$ ): $[0.9, 0.1]$ and $[0.1, 0.9]$ . Now you paste a second copy of the first image, giving three vectors ( $N = 3$ ): $[0.9, 0.1]$ , $[0.9, 0.1]$ , $[0.1, 0.9]$ . Does IS change? Does FID?

Forecast: duplication reduces true diversity. Guess which metric reacts.

Step 1 — IS before. Marginal $[0.5, 0.5]$ . Per-image KL $= 0.9 lo g \frac{0.9}{0.5} + 0.1 lo g \frac{0.1}{0.5} = 0.9 (0.5878) + 0.1 (- 1.6094) = 0.5290 - 0.1609 = 0.3681$ . $IS = exp (0.3681) \approx 1.445$ . Why this step? Establish the clean baseline first, so we can measure the change duplication causes.

Step 2 — IS after (duplicate). New marginal $= \frac{1}{3} ([0.9, 0.1] + [0.9, 0.1] + [0.1, 0.9]) = [0.633, 0.367]$ . Dog images: $0.9 lo g \frac{0.9}{0.633} + 0.1 lo g \frac{0.1}{0.367} = 0.9 (0.3520) + 0.1 (- 1.3002) = 0.3168 - 0.1300 = 0.1868$ . Cat image: $0.1 lo g \frac{0.1}{0.633} + 0.9 lo g \frac{0.9}{0.367} = 0.1 (- 1.8452) + 0.9 (0.8971) = - 0.1845 + 0.8074 = 0.6229$ . $\overline{KL} = \frac{0.1868 + 0.1868 + 0.6229}{3} = 0.3322$ , $IS = exp (0.3322) \approx 1.394$ . Why this step? The duplicate tilts the marginal off uniform ( $0.633$ vs $0.367$ ), and that tilt is precisely what shrinks the diversity term.

Step 3 — reason about FID. FID compares distributions; duplicating a sample shifts $μ_{g}$ and $Σ_{g}$ toward the duplicated point, so FID also moves (rises, since the generated cloud now over-represents one mode). Why this step? Both metrics see the batch's empirical statistics, so both react — IS via a tilted marginal, FID via a shifted mean/covariance.

Verify: IS dropped $1.445 \to 1.394$ ✓ — duplication is punished as lost diversity. Sanity check the direction: fewer distinct modes ⇒ lower IS, higher FID. Consistent.

Recall Quiz yourself

What does $K$ stand for in these formulas? ::: The number of classes the classifier can output; a probability vector has $K$ entries summing to $1$ . Which cell shows IS being fooled by mode collapse in realistic conditions? ::: Cell B2 — slight per-image jitter breaks marginal $=$ conditional, so IS climbs above $1$ even though diversity is dead. What is the FID of two identical Gaussians? ::: $0$ (Cell E) — the floor; centre gap and trace term both cancel. If only the covariances differ, which FID term fires? ::: The trace term $Tr (Σ_{r} + Σ_{g} - 2 (Σ_{r} Σ_{g})^{1/2})$ (Cell G); the mean gap is $0$ . A zero-variance generator against real spread $Σ_{r}$ gives what FID? ::: $Tr (Σ_{r})$ (Cell H), because the square-root cross term vanishes. Why does the perfect IS equal the number of classes $K$ ? ::: Because the exponent maxes out at $H (y) = lo g K$ (uniform marginal) with $H (y ∣ x) = 0$ , and $exp (lo g K) = K$ . IS floor value and FID floor value? ::: IS floor $= 1$ ; FID floor $= 0$ .

Connections

Parent: Evaluating generative models (FID, IS)
Mode Collapse in GANs — the failure IS hides (Cells B2, I) and FID/Recall expose.
Precision and Recall for Generative Models — decomposes what IS/FID lump together.
Inception-v3 Architecture — source of the $p (y ∣ x)$ and 2048-dim features.
Perceptual Loss Functions — why feature space beats pixel space (Cells E–H).

Worked examples — Evaluating generative models (FID, IS)