Visual walkthrough — GAN training instability and mode collapse

This is the picture-story behind the central result of the parent topic. We build everything from zero: what a "distribution" looks like as a picture, what the discriminator is actually deciding, why one specific formula falls out, and — the punchline — why a discriminator that wins too hard silently kills the generator's learning signal.

If you have never seen the GAN setup, read line one and keep going — nothing is assumed.

---

Step 1 — Two piles of dots on a line

WHAT. Forget networks for a moment. Imagine every possible image squished down to a single number $x$ on a horizontal line. A distribution is just "how often each value of $x$ shows up", drawn as a curve whose height at $x$ tells you how likely that $x$ is.

We have two curves:

• $p_{\text{data}}(x)$ — the real data. Where it is tall, real samples cluster.
• $p_g(x)$ — what the generator currently produces.

WHY. GAN training is a fight between these two curves. Everything — instability, mode collapse, the magic $\frac{1}{2}$ — is visible once you can see the two piles overlapping.

PICTURE. Look at the two humps below. The magenta hump is real data; the violet hump is the generator. Where they overlap (the shaded region) is exactly where the discriminator will struggle.

---

Step 2 — What the discriminator is actually deciding

WHAT. Pick one point $x$ on the line. At that spot, two curves have heights $p_{\text{data}}(x)$ and $p_g(x)$. The discriminator $D$ must output a single number $D(x)\in [0,1]$:

$\underset{\text{output}}{\underbrace{D(x)}}=\text{"my belief that this }x\text{ is }\underset{\text{not fake}}{\underbrace{\text{real}}}\text{"}$

• $D(x)=1$ → "surely real"
• $D(x)=0$ → "surely fake"
• $D(x)=0.5$ → "no idea, coin flip"

WHY. The discriminator can only look at the height ratio of the two curves at $x$. If real is much taller there, it should shout "real"; if fake is much taller, "fake". This local view is the seed of the whole derivation.

PICTURE. At one chosen $x$ (the dashed vertical line), we mark the two heights. The discriminator's job is to turn those two heights into one belief.

---

Step 3 — Writing down the score $V$ and reading it as a picture

WHAT. The discriminator is scored by

$V(D,G)=\underset{\text{reward for calling real things real}}{\underbrace{{\mathbb{E}}_{x\sim p_{\text{data}}}[\log D(x)]}}+\underset{\text{reward for calling fakes fake}}{\underbrace{{\mathbb{E}}_{x\sim p_g}[\log (1-D(x))]}}$

Let's decode every symbol:

• ${\mathbb{E}}_{x\sim p_{\text{data}}}[\cdot ]$ — the expectation: "average this quantity, weighting each $x$ by how often real data lands there". It is literally area under (curve × quantity).
• $\log D(x)$ — big (near $0$) when $D(x)\to 1$, and hugely negative when $D(x)\to 0$. So the discriminator is rewarded for pushing $D$ up on real data.
• $\log (1-D(x))$ — the mirror image: rewarded for pushing $D$ down on fakes.

WHY $\log$ and not just $D$? Because $\log$ punishes confidence in the wrong direction infinitely hard. Saying "definitely fake" ($D=0$) about a real image costs $\log 0=-\infty$. This forces honest probabilities, not reckless guesses. No other simple function has this "infinite penalty at the edge" behaviour.

PICTURE. Two reward curves: $\log D$ climbing toward $D=1$, and $\log (1-D)$ climbing toward $D=0$. The discriminator wants to sit high on both — impossible where the curves overlap, which is the tension.

---

Step 4 — Solving one point at a time: the tug-of-war

WHAT. At a single $x$, ignore the averaging and just look at the quantity being summed:

$\ell (d)=\underset{a}{\underbrace{p_{\text{data}}(x)}}\log d+\underset{b}{\underbrace{p_g(x)}}\log (1-d),d=D(x)$

Call the two heights $a$ and $b$. We want the $d$ that makes $\ell$ largest. Slide $d$ from $0$ to $1$: near $0$ the first term is $-\infty$, near $1$ the second term is $-\infty$. So the maximum sits somewhere in the middle, pulled toward whichever height is bigger.

Take the derivative and set it to zero (this is just "find the top of the hill"):

$\frac{d\ell }{dd}=\frac{a}{d}-\frac{b}{1-d}=0⟹a(1-d)=bd⟹\boxed{d^{\ast }=\frac{a}{a+b}}$

WHY a derivative? Because "find the value that maximises a smooth curve" is exactly the question a derivative answers: the top of a hill is where the slope is flat ($=0$). No searching needed.

PICTURE. The little hill $\ell (d)$ over $d\in [0,1]$, with its peak marked. Move the peak marker as $a$ vs $b$ changes.

---

Step 5 — The three cases, seen as heights

WHAT. Now read $D^{\ast }(x)=\frac{a}{a+b}$ across every possible situation:

Region	heights	$D^{\ast }(x)$	meaning
real-only	$a\gg b$	$\to 1$	"obviously real"
fake-only	$a\ll b$	$\to 0$	"obviously fake"
perfect match	$a=b$	$=\frac{1}{2}$	"pure coin flip"
empty region	$a=b=0$	$\frac{0}{0}$ undefined	see Step 6

WHY these cases matter. The whole promise of GANs is: when the generator perfectly copies the data, the game reaches a draw. Case 3 is that draw — if $p_g=p_{\text{data}}$ everywhere then $D^{\ast }=\frac{1}{2}$ everywhere, and the discriminator is reduced to guessing. That is the target we want training to reach.

PICTURE. The two humps again, colour-coded by what $D^{\ast }$ does across the line: pure magenta zone $\to 1$, pure violet zone $\to 0$, overlap fading to the grey $\frac{1}{2}$ line.

---

Step 6 — The degenerate case: where the curves don't touch

WHAT. Early in training the generator's hump sits far away from the real hump — no overlap. In that gap, $p_{\text{data}}=0$ and $p_g=0$, so $D^{\ast }=\frac{0}{0}$: undefined. But just left of the fake hump and just right of the real hump, one curve is zero and the other is not, so $D^{\ast }$ snaps to exactly $0$ or exactly $1$.

WHY this is the danger zone. A discriminator that can output a perfect step — $1$ on real, $0$ on fake, with a cliff in between — has zero slope everywhere except the cliff. The generator's samples live in the flat $D=0$ plateau, where the surface it's trying to climb is perfectly flat.

PICTURE. Non-overlapping humps and the resulting $D^{\ast }$ curve: a flat shelf at $0$, a vertical cliff, a flat shelf at $1$. Mark that the fake samples sit on the flat shelf.

---

Step 7 — Where the gradient dies (the punchline)

WHAT. The generator improves by nudging its samples "uphill" on the discriminator's surface. Its gradient contains the factor ${\nabla }_{x}D(G(z))$ — how steeply $D$ rises as you move the sample. On the flat plateau of Step 6:

$\underset{\text{constant prefactor}}{\underbrace{(-1)}}\cdot \underset{\to 0\text{(flat!)}}{\underbrace{{\nabla }_{x}D(G(z))}}\cdot {\nabla }_{\theta }G(z)⟶0$

• ${\nabla }_{x}D$ — the slope of the discriminator surface at the sample. On a flat shelf this is $0$.
• The prefactor $-1$ (from $\log (1-D)$ at $D\approx 0$) is not the problem — it stays a healthy constant.
• Multiply anything by a zero slope → zero learning signal.

WHY this is the real culprit. The generator isn't stuck because "the discriminator is confident" in some vague sense — it's stuck because a saturated discriminator gives a flat surface, and you cannot roll uphill on a flat floor. See why flat regions kill gradient descent.

PICTURE. A ball (a fake sample) sitting on a flat plateau of the $D$ surface, with a sad arrow showing "nowhere to roll". Contrast with a tilted surface where the ball rolls toward real data.

---

Step 8 — The fix, as a re-tilted surface

WHAT. Two rescues, both visible as "give the generator back a slope":

1. Non-saturating loss: maximise $\log D(G(z))$ instead of minimising $\log (1-D(G(z)))$. Now the prefactor is $\frac{1}{D(G(z))}$, which blows up ($1/0.001=1000$) exactly when the surface is flattest — amplifying the tiny slope back into a usable signal.
2. Wasserstein critic: drop the sigmoid entirely. The critic $f_w(x)$ outputs a raw number with a guaranteed non-zero slope (1-Lipschitz), so the surface is never flat — the ball always has somewhere to roll.

WHY it works. Both change the shape of the surface the generator climbs, not the fixed point it's aiming for. Same target ($p_g\to p_{\text{data}}$), gentler hills.

PICTURE. Side by side: the dead flat plateau (original loss) vs. the re-tilted, always-sloped surface (fixes), same ball, now rolling.

---

The one-picture summary

Everything on one canvas: two humps → the height ratio → $D^{\ast }=\frac{a}{a+b}$ → three cases → the flat plateau where gradient dies → the re-tilt that saves it.

Recall Feynman retelling — say it back in plain words

Picture two piles of dots on a line: the real pile and the fake pile the generator makes. The discriminator stands at each spot and asks "of the dots here, what fraction are real?" — that fraction is the best it can possibly answer, $\frac{a}{a+b}$. When the two piles land perfectly on top of each other, that fraction is $\frac{1}{2}$ everywhere: the discriminator is forced to flip coins. That's the draw we want. But early on the piles are far apart, so the discriminator can draw a perfect cliff — flat $0$ over the fakes, flat $1$ over the reals. The generator's fakes sit on the flat $0$ floor, and to improve it needs the floor to tilt toward the real pile. A flat floor means no direction to walk — that's the dead gradient. The saturated prefactor isn't the villain; the flat floor is. The fixes (non-saturating loss, Wasserstein critic) all do one thing: tilt the floor back so the generator can walk home.

Recall Quick self-test

Why is $D^{\ast }(x)=\frac{1}{2}$ everywhere the sign of success? ::: It means $p_g=p_{\text{data}}$ at every point, so real and fake are indistinguishable — the generator has matched the data. What term actually vanishes in the dead-gradient case, and why? ::: ${\nabla }_{x}D\to 0$, because a saturated discriminator's output is flat away from the decision cliff. Why does maximising $\log D(G(z))$ rescue training? ::: Its prefactor $\frac{1}{D(G(z))}$ grows large exactly when $D(G(z))\to 0$, amplifying the small slope back into a usable signal.

---

Connections

• Parent: GAN training instability and mode collapse
• Hinglish: 4.5.08 GAN training instability and mode collapse (Hinglish)
• Prereq: 4.5.01-Generative-Adversarial-Networks-fundamentals
• The fix in depth: 4.5.07-Wasserstein-GAN-and-improved-training
• Flat-region intuition: 3.4.05-Optimization-challenges-in-deep-learning
• Related stabilisers: 4.3.06-Batch-normalization-and-alternatives, 4.5.09-StyleGAN-and-progressive-growing, 4.5.10-Conditional-GANs-and-control