Evaluating generative models (FID, IS): Visual walkthrough

This page builds the Fréchet Inception Distance from nothing. We will not use a single symbol before we have drawn it. By the end you will see why the formula looks the way it does, and why it catches problems that the Inception Score misses.

We are answering one question the whole way: "How far apart are two clouds of points?" — the cloud of real images and the cloud of fake images.

Step 1 — Turn every image into a single dot

WHAT. We take one image and squeeze it down to a short list of numbers — a feature vector. Think of it as the image's coordinates in a "meaning space".

WHY. Comparing raw pixels is hopeless: shift a cat one pixel to the right and every pixel value changes, yet the picture is identical to your eye. We want a space where perceptually similar images sit close together. The Inception-v3 Architecture gives us exactly this — its second-to-last layer (called pool3) outputs a vector of 2048 numbers that encode edges, textures and object-parts rather than raw brightness. This is the same idea behind Perceptual Loss Functions.

PICTURE. Below, an image goes into a box (the network) and comes out as one dot placed somewhere in a plane. We only draw 2 dimensions so a human can see it — the real space has 2048, but every idea on this page works the same in any number of dimensions.

Figure — Evaluating generative models (FID, IS)

Step 2 — Two clouds of dots: real vs fake

WHAT. Run many real images through the box → a cloud of blue dots. Run many generated images through the box → a cloud of orange dots.

WHY. A generative model is good when its fake cloud sits on top of the real cloud — same centre, same shape, same spread. If the orange cloud is shifted, or squished, or only covers a corner of the blue cloud, the model is failing. So "quality of the model" becomes a geometry question: how different are these two clouds?

PICTURE. Two overlapping blobs. Notice the orange blob is a bit off-centre and a bit differently shaped — that mismatch is exactly what FID must measure.

Step 3 — Describe each cloud with a centre and a spread

WHAT. Instead of storing every dot, we summarise each cloud with just two things:

its centre (the average dot), written $μ$ (Greek "mu"),
its spread and tilt, written $Σ$ (Greek capital "Sigma").

WHY. Comparing millions of dots to millions of dots is a nightmare. But if a cloud looks like a smooth elliptical blob, then its centre and its spread describe it completely. We are modelling each cloud as a Gaussian (a bell-shaped blob). This is an approximation — but a cheap, closed-form one, and it is exactly what makes FID computable.

PICTURE. The blue cloud with its centre $μ_{r}$ marked as a cross, and an ellipse drawn around it showing $Σ_{r}$ — how far the dots spread and which way the blob leans.

We now have four summaries: $real (blue) μ_{r}, Σ_{r} generated (orange) μ_{g}, Σ_{g}$ The subscript $r$ means real, $g$ means generated.

Step 4 — Part one of the distance: how far apart are the centres?

WHAT. Measure the straight-line distance between the two centres, then square it.

WHY. The most obvious way two clouds differ is location: the orange blob might sit off to one side. That is a mean shift, and it maps to a real failure — e.g. all fakes look slightly too dark, or too smooth. We square the distance because squared distance is what the underlying maths (next step) naturally produces, and it makes the term grow fast as clouds drift apart.

PICTURE. An arrow from $μ_{r}$ to $μ_{g}$ ; its length is what we square.

But two blobs can share a centre and still differ — one could be a tight dot, the other a wide smear. Centres alone are not enough. We need the shapes too.

Step 5 — Part two: how differently shaped are the two blobs?

WHAT. Compare the two spread-ellipses $Σ_{r}$ and $Σ_{g}$ with the term $Tr (Σ_{r} + Σ_{g} - 2 (Σ_{r} Σ_{g})^{1/2}) .$

WHY, piece by piece.

$Σ_{r} + Σ_{g}$ — add up the total "amount of spread" in both blobs.
$(Σ_{r} Σ_{g})^{1/2}$ — a kind of geometric mean of the two shapes. It is large only when the two ellipses point the same way and are the same size. It is our "how much do the shapes agree" reward.
The $- 2$ subtracts twice that agreement. So if the shapes are identical, the reward exactly cancels the total spread and this whole bracket collapses to $0$ .
$Tr (\cdot)$ — the trace, meaning "add up the diagonal entries". It turns the whole grid of numbers into a single number by summing the mismatch across all 2048 directions.

PICTURE. Two ellipses of different size and tilt; a shaded region shows the mismatch the trace is measuring. When ellipses match, no shaded region remains.

Step 6 — Snap the two parts together: the FID formula

WHAT. Distance between two clouds = (centre gap) + (shape gap).

WHY. These are the only two ways two Gaussian blobs can differ — location and shape. Add them and you have the full Fréchet distance (the squared 2-Wasserstein distance) between the two Gaussians, which has this exact closed form.

PICTURE. The full formula with each chunk colour-tied to the picture of what it measures.

Where does this closed form come from? The 2-Wasserstein distance $W_{2}$ asks: what is the cheapest way to transport one pile of probability onto the other, cost = squared distance moved? The quantity below is the square of $W_{2}$ — note the $∥ x - y ∥^{2}$ inside the expectation, which is why FID equals $W_{2}^{2}$ and not $W_{2}$ itself: $W_{2}^{2} (p, q) = in f_{π} E_{(x, y) \sim π} [∥ x - y ∥^{2}]$

$π$ — a coupling: a plan pairing each bit of the blue cloud with a bit of the orange cloud.
$in f$ — take the cheapest such plan. For two Gaussians that infimum is solved on paper, and the answer is exactly the two terms above. So precisely: $FID = W_{2}^{2}$ , the squared Wasserstein-2 distance — people often say "FID is the Wasserstein-2 distance" loosely, but the formula is its square.

Step 7 — The degenerate and edge cases (never get surprised)

WHAT / WHY / PICTURE for each corner case:

PICTURE. Four mini-panels, one per case.

Identical clouds. $μ_{r} = μ_{g}$ and $Σ_{r} = Σ_{g}$ . Centre gap $= 0$ ; shape term $= Tr (Σ + Σ - 2 (ΣΣ)^{1/2}) = Tr (2Σ - 2Σ) = 0$ . So FID $= 0$ . The only way to score zero.
Mode collapse. The generator outputs almost the same image every time → its orange cloud shrinks to a tiny dot → $Σ_{g} \approx 0$ . Then the shape term becomes $Tr (Σ_{r})$ , a large positive penalty, and the centre likely shifts too. FID shoots up. This is the big win over IS, which was blind to mode collapse. (See also Precision and Recall for Generative Models for a finer-grained view of coverage.)
Same centre, wrong spread. $μ_{r} = μ_{g}$ but $Σ_{g}$ is too tight or too wide. Centre gap $= 0$ , yet the shape term is positive. FID still flags it — you cannot cheat FID by only matching averages.
Number of samples too small. With few images, $Σ$ is badly estimated (noisy), so FID is biased upward. This is why practice uses 10k–50k images per side. FID is only trustworthy with enough dots.

The one-picture summary

Everything on one canvas: two clouds, the arrow between centres (Step 4), the two ellipses (Step 5), and the assembled formula (Step 6) — with an arrow reminding you low FID = clouds coincide.

Recall Feynman retelling — say it back in plain words

First I shrink every picture down to a dot using Inception-v3, so similar-looking pictures land near each other. Now I have a blue cloud of dots for real images and an orange cloud for fake ones. A good generator's orange cloud should lie right on top of the blue one. I summarise each cloud two ways: where its centre is ( $μ$ ) and how it spreads and tilts ( $Σ$ ). Then FID is just two penalties added together: how far the two centres are apart (squared), and how differently the two blobs are shaped (the trace term, which is zero only when the ellipses match perfectly). Add them: that's the distance between the clouds — precisely the squared Wasserstein-2 distance. Zero means perfect. If the generator collapses to one image, its cloud becomes a dot, the shapes disagree wildly, and FID explodes — which is exactly why FID catches mode collapse when the Inception Score cannot.

Recall Quick self-test

Why square the mean difference? ::: Because the underlying Wasserstein-2 distance uses squared transport cost; it also makes the penalty grow fast as centres drift. What does the trace do in the shape term? ::: Collapses the whole 2048-direction covariance mismatch into a single number by summing the diagonal. When is FID exactly zero? ::: Only when both centres and both covariances are identical. How does FID catch mode collapse but IS doesn't? ::: FID sees the collapsed cloud shrink to a dot ( $Σ_{g} \to 0$ ) and mismatch the real cloud; IS never compares to real data. Lower FID or higher FID = better? ::: Lower — it is a distance. Is FID the Wasserstein-2 distance or its square? ::: Its square — FID $= W_{2}^{2}$ , using cost $∥ x - y ∥^{2}$ inside the expectation. Does FID use the $\frac{1}{N}$ or $\frac{1}{N - 1}$ covariance? ::: $\frac{1}{N}$ (population covariance), matching the original paper and implementations.