Foundations — GAN training instability and mode collapse

This page builds the vocabulary for GAN training instability and mode collapse from nothing. Nothing here assumes you have seen probability notation, expectations, logarithms-as-scores, or gradients before. Every symbol earns its place before we use it.

1. A "distribution" and the symbol $p$

Picture a landscape of hills. Where the hill is tall, that kind of $x$ shows up often. Where the ground is flat and low, that $x$ is rare. The whole area under all the hills adds up to $1$ (something always happens). Written precisely, for a continuous $x$ this "total area is one" rule is

$\int p(x)dx=1\text{(or, for discrete x,}{\sum }_{x}p(x)=1\text{)}.$

We need two of these landscapes:

• $p_{data}(x)$ — the landscape of real data (real photos, real digits). This is fixed; it is the target.
• $p_g(x)$ — the landscape of what the generator currently produces. This one moves during training.

2. The sampling arrow $x\sim p$

Picture throwing a dart at the hill-landscape: you are more likely to hit where the hills are tall. Each dart-hit is one sample $x$. We need this because networks never see the whole landscape — they only ever see a bucket of samples drawn from it.

3. Noise $z$ and the generator $G$

Picture a slot machine: pull a random lever ($z$) and the machine ($G$) prints a picture. Different $z$ → different picture. As $G$ learns, the pictures it prints get more realistic.

• Why noise at all? Randomness is where variety comes from. If $z$ were always the same, $G$ would print one fixed picture. Mode collapse is exactly the failure where different $z$ all get mapped to the same output — the variety in $z$ gets thrown away.

4. The discriminator $D$ and the interval $[0,1]$

Picture a dial the judge turns: hard left = fake, hard right = real, middle = shrug. The judge is good when it turns the dial to $1$ for real photos and to $0$ for the generator's fakes.

5. The logarithm as a "surprise / reward" meter

Why does $\log$ appear all over the objective? Because we want to reward the judge for confident-correct answers and punish confident-wrong ones — and $\log$ does exactly that stretching.

• Why not just use $D$ itself as the score? Compare two nudges of the same size, $0.09$. Measured with plain $D$, moving from $0.90$ to $0.99$ and moving from $0.09$ to $0.18$ look identical — both are "worth $0.09$". Measured with $\log$, the first nudge is worth $\log (0.99)-\log (0.90)\approx 0.095$, while the second is worth $\log (0.18)-\log (0.09)\approx 0.693$ — over seven times more reward. So $\log$ deliberately makes changes near certainty count for very little and changes near ignorance count for a lot.

6. Expectation $\mathbb{E}$ — the "average over the bucket"

Picture: score every dart-hit on the hill, then average all the scores — but hits in the tall regions count more because there are more of them (that is exactly the weight $p(x)$ inside the integral). We need $\mathbb{E}$ because the objective must judge $D$ and $G$ over the whole landscape, not one lucky sample.

7. The full objective, now readable

With every symbol built, the parent's headline formula is now plain English:

Here $1-D(G(z))$ is the judge's belief that the fake is fake; $\log$ of it rewards the judge for spotting fakes, and $G$ wants that reward to be small.

8. Gradients — the "which way to step" arrow, and who steps which way

Picture standing on a foggy hillside feeling for the steepest slope with your foot, then stepping down. That's gradient descent.

We care about all this because "$G$ receives no learning signal" — the heart of instability — literally means the arrow ${\nabla }_{\theta }{L}_G$ has near-zero length, so the foot feels flat ground and can't decide which way to step.

• ${\nabla }_{x}D$ (gradient with respect to the input $x$, not the weights) measures how much the judge's verdict changes if you nudge the picture. When the judge is over-confident its verdict is flat everywhere, so ${\nabla }_{x}D\to 0$ — no nudge changes its mind, so $G$ gets no hint. That is the vanishing gradient in one line, exactly the saturation warned about in Section 5.

Prerequisite map

Each foundation feeds the next: landscapes → sampling → the two players → their shared score → the arrows that move them → the failures the topic is about.

Where this connects

• The whole GAN setup comes from 4.5.01-Generative-Adversarial-Networks-fundamentals.
• The "flat foot / no downhill" gradient trouble is a general theme in 3.4.05-Optimization-challenges-in-deep-learning.
• The cleaner-score fix that replaces $\log$ lives in 4.5.07-Wasserstein-GAN-and-improved-training.
• Once stable, richer architectures follow in 4.5.09-StyleGAN-and-progressive-growing and 4.5.10-Conditional-GANs-and-control.
• Layer statistics that also affect stability: 4.3.06-Batch-normalization-and-alternatives.
• Hinglish version: 4.5.08 GAN training instability and mode collapse (Hinglish).

Equipment checklist