Evaluating generative models (FID, IS): Question bank

This page is a conceptual stress-test for the parent topic on FID and IS. Nothing here needs a calculator — every item targets a misunderstanding or a boundary case that these two metrics quietly invite. Read the prompt, answer out loud, then reveal.

Before we start, three plain-word reminders so no symbol appears unearned:

True or false — justify

A higher Inception Score always means a better generative model.

False. IS only rewards confident, varied class labels; it never looks at real data, so a model producing sharp-but-fake garbage or exactly one image per class can score high while being useless.

FID of 0 means the generator has perfectly matched the real image distribution.

True in the Gaussian model —

μ_{r} = μ_{g}

and

Σ_{r} = Σ_{g}

forces both terms to zero — but because FID assumes Gaussians, FID 0 really means the first two moments match, not necessarily the full distribution.

FID can be negative.

False. It is a squared distance plus a trace of a positive-semidefinite difference; both pieces are

\geq 0

, so FID is bounded below by 0.

Inception Score can be negative.

False. IS is

exp (something)

, and the exponential of any real number is positive, so IS

> 0

always, with the floor near 1 for a useless model.

Lower FID and higher IS both mean "better".

True, but the directions differ: FID is a distance (smaller = closer to real), IS is an exponentiated divergence (larger = more confident-and-diverse). Confusing their directions is a classic slip.

FID needs labelled real images to compute.

False. FID only uses Inception features (the 2048-dim fingerprint), not the class labels, so no labels are required — unlike IS, which relies on the classifier's label probabilities.

If you double the number of generated samples, the true FID roughly doubles.

False. FID estimates the same underlying distance; more samples reduce estimation bias/variance (FID is biased upward for small

N

) but the value converges, it does not scale with

N

IS is comparable across datasets — an IS of 8 on CIFAR-10 and an IS of 8 on ImageNet mean the same thing.

False. IS is capped by the number of classes the classifier knows and by how those classes fit the data, so it has no absolute cross-dataset scale.

Spot the error

"My GAN collapsed to a single perfect dog image, so its Inception Score must be terrible." — where's the error?

The confident

p (y ∣ x)

still gives low conditional entropy, and if the model outputs only dogs the KL per sample can stay large; IS is often blind to this mode collapse — you need FID to catch it.

"FID compares images pixel by pixel after aligning them, that's why it's perceptual." — fix it.

Wrong mechanism: FID never touches pixels. It compares Gaussian summaries of Inception features, which is what makes it perceptual — a one-pixel shift barely changes the features.

"To compute

(Σ_{r} Σ_{g})^{1/2}

I just take the square root of every entry of the matrix product." — spot the mistake.

The matrix square root is not element-wise; it is

U Λ^{1/2} U^{T}

from the eigendecomposition, the matrix

M

such that

M M = Σ_{r} Σ_{g}

"IS =

exp (H (y) - H (y ∣ x))

, so I want both entropies as large as possible." — correct it.

You want

H (y)

large (diverse labels) but

H (y ∣ x)

small (confident per-image predictions); maximising both would cancel and shrink the score.

"FID's mean term

∥ μ_{r} - μ_{g} ∥^{2}

already captures everything, the trace part is redundant." — why is this wrong?

The mean term only measures where the clouds sit; two clouds can share a centre yet differ wildly in spread and shape — that mismatch lives entirely in the covariance/trace term.

"Since Inception-v3 was trained on ImageNet, FID works equally well on faces, medical scans, and cartoons." — the error?

Out-of-domain images push Inception into feature regions it never learned, so the features (and thus FID) become unreliable — a known limitation when the data is far from ImageNet.

Why questions

Why exponentiate the KL average to get IS instead of leaving it in log-space?

To turn it into an "effective number of classes" scale where 1 ≈ useless and up to (number of classes) ≈ ideal, making the number human-interpretable.

Why model the feature clouds as Gaussians for FID when they clearly aren't perfectly Gaussian?

Because a Gaussian is fully described by a mean and covariance, and only then does the 2-Wasserstein distance collapse to a closed-form expression — a tractable approximation, not a claim of true normality.

Why does IS use the marginal

p (y)

as the reference inside the KL, rather than a fixed uniform distribution?

Using the model's own average

p (y)

makes the KL simultaneously reward per-image confidence and penalise a lopsided label mix; a fixed uniform would ignore what the model actually produces overall.

Why is FID called a "distance" but IS is called a "score"?

FID literally measures how far two distributions are (a metric-like quantity, real vs. fake), while IS is a self-contained quality-plus-diversity summary of the generated set alone with no comparison target.

Why prefer Inception's penultimate (pool3) layer over its final softmax outputs for FID?

The penultimate layer holds rich 2048-dim semantic features; the softmax has already compressed everything into a few class scores, throwing away the fine structure FID needs to compare distributions. Compare this idea to perceptual losses, which also read intermediate features.

Why do FID and IS often disagree, and which do we trust more for mode collapse?

IS can stay high under collapse because it never sees real data; FID rises when generated diversity drops because the fake covariance shrinks away from the real one — so FID is the more reliable collapse detector. For a finer split of quality vs. coverage, see Precision and Recall for Generative Models.

Edge cases

What is the IS of a generator that outputs the exact same confident image every time?

Per-image

p (y ∣ x)

is confident, but

p (y)

equals that same peaked list, so

p (y ∣ x) = p (y)

, the KL is 0, and IS

= exp (0) = 1

— the "useless" floor.

What is the IS of a generator producing pure noise that the classifier labels uniformly?

Each

p (y ∣ x)

is near-uniform (high conditional entropy), so the KL against the uniform

p (y)

is near 0 and IS

\approx 1

again — noise and mode collapse both bottom out at 1 for opposite reasons.

What happens to FID if real and generated features are identical distributions?

μ_{r} = μ_{g}

kills the mean term, and

Σ_{r} = Σ_{g}

makes

Σ_{r} + Σ_{g} - 2 (Σ_{r} Σ_{g})^{1/2} = 2Σ - 2Σ = 0

, so FID

= 0

— the perfect-match limit.

What if the covariance matrices are singular (some feature directions have zero variance)?

The matrix square root

(Σ_{r} Σ_{g})^{1/2}

is still defined for positive-semidefinite matrices (zero eigenvalues just give zero square-root eigenvalues), but numerically sqrtm may return tiny imaginary parts that must be discarded.

What is FID when you compute it between a real set and itself using two different random halves?

It is small but not exactly 0, because finite-sample estimates of

μ

and

Σ

differ slightly between halves — this "FID floor" is why you never expect a real generator to reach literally 0.

How does IS behave as the number of samples

N

shrinks toward a handful of images?

The marginal

p (y)

becomes a poor estimate and the score grows unstable and biased, which is why standard practice uses tens of thousands of samples (e.g. 50,000).

What does a very low

H (y ∣ x)

but also very low

H (y)

tell you about the model?

Images are confidently classified (good sharpness) yet all fall into few classes (poor diversity) — the signature of partial mode collapse, and the KL average stays small so IS is low.

Recall Fast self-check

IS blind spot ::: it ignores real data, so it can't see mode collapse or non-realism. FID lower bound ::: 0, achieved only when both mean and covariance match. The one shared tool ::: both metrics read Inception-v3 features rather than pixels.