Foundations — Evaluating generative models (FID, IS)

This page assumes you know nothing about the notation on the parent note. We build every symbol from the ground up, in an order where each one leans only on the ones before it.

0. The cast of characters (what a "distribution" even is)

Before any formula, picture a giant bin of photographs.

Look at the two clouds above. The teal cloud is real images; the orange cloud is what a generator produces. Every dot is one image. Two things can go wrong:

• the orange cloud sits in the wrong place (fake images look different from real ones), or
• the orange cloud is the wrong shape/size (too tight = not diverse, too spread = sloppy).

FID measures exactly these two failures. Keep this picture in your head for the whole page.

1. Symbols for "a sample" and "the generator"

Picture: reaching blindfolded into the orange bin and pulling out one photo. That act is $x\sim p_g$.

Why the topic needs it: FID compares $p_r$ against $p_g$; IS studies $p_g$ alone. You cannot state either goal without names for "the real cloud" and "the fake cloud".

2. The averaging symbol: expectation $\mathbb{E}$

Almost every formula ahead says "average this over many images." We give that its own symbol now, before it is used.

Why the topic needs it: $p(y)$, the average-KL inside IS, and the parameters $\mu ,\Sigma$ are all expectations. Having $\mathbb{E}$ defined up front means no later formula sneaks in an undefined symbol.

3. From an image to a list of numbers: features

A raw image is a huge grid of pixel brightnesses. Comparing pixels directly is useless — shift a cat one pixel left and every pixel changes, yet it's the same cat. So we first translate the image into meaningful numbers.

Think of each image as an arrow into a 2048-dimensional room (we draw it in 2D — same idea, fewer axes). Perceptually similar images land close together; a cat and a car land far apart. This is why we say Inception is a "learned perceptual judge" — closeness of arrows matches closeness to the human eye, unlike raw pixels.

Why the topic needs it: both FID and IS run on these feature vectors, never on raw pixels. Perceptual Loss Functions use the same trick for the same reason.

4. Mean $\mu$ — the centre of a cloud

The picture: ${\mu }_r$ is the centre of the teal cloud, ${\mu }_g$ the centre of the orange cloud. If the two centres are far apart, the generator is systematically off. That is the first term of FID: $\Vert {\mu }_r-{\mu }_g{\Vert }^2$ (the squared distance between the two centres — Section 7 explains the $\Vert \cdot \Vert$).

5. Covariance $\Sigma$ — the shape and spread of a cloud

The mean tells you where a cloud sits, not how big or which way it stretches. For that we need covariance.

The figure shows three clouds with the same $\mu$ but different $\Sigma$: a small ball (low diversity), a large ball (high diversity), and a tilted ellipse (features correlated). Note: $\Sigma$ being big is not automatically good — you want the generated ${\Sigma }_g$ to match the real ${\Sigma }_r$, not exceed it. A cloud that is too tight is the visual signature of Mode Collapse in GANs.

Why the topic needs it: FID's second term compares ${\Sigma }_r$ and ${\Sigma }_g$, catching the mode collapse that IS is blind to.

6. Modelling clouds as Gaussians $\mathcal{N}(\mu ,\Sigma )$

The picture: a fuzzy elliptical mound, densest at $\mu$, fading outward with a spread set by $\Sigma$.

Why the topic needs it: comparing two arbitrary clouds is impossibly hard. But if you assume both are Gaussians, the distance between them has a clean formula (Section 8). FID makes exactly this simplifying assumption.

7. Distance symbols: $\Vert \cdot {\Vert }^2$ and $\text{Tr}$

Two clouds differ in centre and in shape. We need one number for each gap.

Why the topic needs both: FID adds a centre gap ($\Vert {\mu }_r-{\mu }_g{\Vert }^2$) to a shape gap (a trace expression) to get its final single score.

8. Matrix square root $({\Sigma }_r{\Sigma }_g{)}^{1/2}$

The FID formula contains a strange-looking object: the square root of a matrix. We build it from the plain-number version.

Picture: a matrix takes a circle of arrows and stretches it into an ellipse. Its square root is the "half-strength" version — the stretch you apply twice to get the full ellipse. In FID, $({\Sigma }_r{\Sigma }_g{)}^{1/2}$ is a geometric average of the two cloud shapes: it is large only when the real and fake shapes stretch in the same directions, so subtracting $2\times$ it rewards matching shapes.

9. Entropy $H$ and KL divergence — the tools behind IS

IS never looks at clouds of features; it looks at classifier guesses. Different symbols, same classifier.

See Precision and Recall for Generative Models for an alternative that separates quality and diversity into two numbers instead of merging them like IS does.

10. The Fréchet / Wasserstein-2 distance = FID

That whole expression is FID, once $\mu ,\Sigma$ come from Inception features. So FID = "Fréchet distance between the Gaussian-modelled real and fake feature clouds," and lower is better.

Prerequisite map

Notice the two independent branches feeding the topic: the feature/cloud branch builds FID, the classifier-guess branch builds IS. They share only the Inception network and the expectation symbol.

Equipment checklist

Read each question, answer in your head, then reveal.