GAN training instability and mode collapse: Worked examples

This page is the drill hall for the parent topic. The parent built the theory: the minimax game, the optimal discriminator, the vanishing-gradient fix, and mode collapse. Here we run numbers through every corner case so you never meet a scenario you haven't seen worked.

Before we start, one promise: we re-anchor every symbol you'll actually use on this page — including the two the parent leans on heavily, $\nabla_{x} D$ and the optimal game value $V^{*}$ .

Definition The quantities we plug numbers into

$D (x)$ ::: the discriminator's guess, a number between $0$ and $1$ , that sample $x$ is real. $D = 1$ means "surely real", $D = 0$ means "surely fake", $D = 0.5$ means "coin flip".
$p_{d a t a} (x)$ and $p_{g} (x)$ ::: how often the real world (data) versus the generator produces sample $x$ . Think of them as two histograms over the same shelf of samples.
$D^{*} (x) = \frac{p _{d a t a} ( x )}{p _{d a t a} ( x ) + p _{g} ( x )}$ ::: the best possible discriminator when both histograms are known — it just reports "what fraction of the stuff at $x$ is real?"
$θ$ ::: the generator's knobs — the list of numbers (network weights) that control what pictures $G$ makes. Turning these knobs changes $G (z)$ .
$\nabla_{θ} G (z)$ ::: "how the generated sample moves when we nudge each knob $θ$ ." It is the collection of tiny sensitivities of the output to every weight — the bridge that turns "change the picture" into "change the knobs".
$\nabla_{x} D (x)$ ::: the slope of the discriminator at the sample $x$ — "if I nudge the sample itself a hair, how much does $D$ 's score change?" The subscript $x$ means we differentiate with respect to the input sample (not the weights). On the S-curve of Case D it is the steepness of the line touching the curve: steep in the middle, nearly flat at the confident ends. This is the factor that dies when $D$ saturates.
$V^{*}$ ::: the game value once the discriminator is optimal, i.e. we plug $D^{*}$ back into the GAN objective and read off the resulting number. In Ex 1 we compute it per sample as $lo g D^{*} + lo g (1 - D^{*})$ ; $V^{*}$ is that quantity totalled over all samples. It is the single number that measures "how well is the generator doing at the best judge's mercy?".

The scenario matrix

Every case this topic can throw at you falls into one of these cells. The 8 worked examples below are each stamped with the cell(s) they cover.

Cell	The situation	Where numbers get weird	Example
A. Perfect generator	$p_{g} = p_{d a t a}$ everywhere	$D^{*} = 0.5$ , game is a draw	Ex 1
B. One-sided region	$p_{d a t a} > 0$ but $p_{g} = 0$ (a mode the generator missed)	$D^{*} = 1$ , a "hole"	Ex 2
C. Over-produced region	$p_{g} > 0$ but $p_{d a t a} = 0$ (garbage output)	$D^{*} = 0$	Ex 2
D. Saturated-D gradient	$D (G (z)) \approx 0$ , confident critic	input-gradient $\nabla_{x} D \to 0$ : signal vanishes	Ex 3
E. Loss swap	compare $lo g (1 - D)$ vs $lo g D$ gradient scale	prefactor $- 1$ vs $1/ D \to \infty$	Ex 4
F. Mode-collapse incentive	$D$ favours old mode over new mode	loss gap $L_{G} (x_{2}) - L_{G} (x_{1})$	Ex 5
G. Oscillation / non-convergence	$D$ forgets modes, $G$ cycles	period-2 orbit, no fixed point	Ex 6
H. Real-world word problem	JS-divergence at optimum	value collapses to $- lo g 4$	Ex 7
I. Exam twist (WGAN)	critic $f_{w}$ , Lipschitz bound	Wasserstein distance stays finite where JS blows up	Ex 8

Case A & the optimal discriminator

Worked example

Ex 1 — Perfect generator (Cell A) The generator has fully matched the data: at every sample $x$ , $p_{d a t a} (x) = p_{g} (x) = 0.4$ . What does the optimal discriminator output, and what is the game value there?

Forecast: Guess before reading — if the fake is a flawless copy of the real, can any judge do better than a coin flip?

Step 1. Plug into $D^{*} (x) = \frac{p _{d a t a}}{p _{d a t a} + p _{g}} = \frac{0.4}{0.4 + 0.4}$ . $D^{*} (x) = \frac{0.4}{0.8} = 0.5$ Why this step? $D^{*}$ is literally "fraction of stuff here that is real". Half the stuff is real, half is generated → $0.5$ .

Step 2. Compute the per-sample value $lo g D^{*} + lo g (1 - D^{*})$ — this is $V^{*}$ evaluated at one sample: $lo g (0.5) + lo g (0.5) = 2 lo g (0.5) = - 1.386$ Why this step? The GAN objective at the optimum equals this constant everywhere; it is $- lo g 4 = - 1.386$ , the theoretical floor.

Verify: $- lo g 4 = - 1.3863$ . Matches. The discriminator is reduced to guessing — exactly the "draw" we expect from a perfect generator. Sanity: $D^{*} = 0.5$ means zero useful gradient, which is why training near convergence is fragile.

Cases B & C — the holes and the garbage

Figure — GAN training instability and mode collapse

The figure shows two histograms on one shelf: blue is $p_{d a t a}$ , orange is $p_{g}$ . The three vertical markers pin the exact points used in Ex 2: the green dashed line at $x_{A}$ (both curves overlap — they agree), the red dashed line at $x_{B}$ (only blue is present — the generator's hole), and the orange dashed line at $x_{C}$ (only orange is present — the generator's garbage). Each marker is labelled with its $(p_{d a t a}, p_{g})$ pair so you can read the numbers straight off the picture.

Worked example

Ex 2 — Missed mode and over-produced garbage (Cells B & C) At three locations we measure:

$x_{A}$ : $p_{d a t a} = 0.6, p_{g} = 0.6$ (both agree — the green marker in the figure)
$x_{B}$ : $p_{d a t a} = 0.5, p_{g} = 0.0$ (a mode the generator missed — the red marker)
$x_{C}$ : $p_{d a t a} = 0.0, p_{g} = 0.5$ (the generator makes stuff that never occurs in reality — the orange marker)

Find $D^{*}$ at each.

Forecast: Where the generator produces nothing real-looking that's actually real, how confident should a perfect judge be?

Step 1 — agree point $x_{A}$ (green marker): $D^{*} = \frac{0.6}{0.6 + 0.6} = 0.5$ . Why? Same logic as Ex 1 — locally balanced, coin flip. On the figure both curves sit on top of each other here.

Step 2 — the hole $x_{B}$ (red marker): $D^{*} = \frac{0.5}{0.5 + 0} = 1$ . Why this step? Everything at $x_{B}$ is real (the generator never comes here — orange curve is on the floor). A perfect judge is certain: this is real. This certainty is the danger — the generator gets a strong "you should come here" gradient only if the loss geometry lets that signal flow. In vanilla GANs it often does not; the mode stays empty. This is the fingerprint of mode collapse.

Step 3 — the garbage $x_{C}$ (orange marker): $D^{*} = \frac{0}{0 + 0.5} = 0$ . Why? Everything here is generator junk, no real data (blue curve is on the floor). Perfect judge says "fake", certainty $0$ .

Verify: $D^{*} \in {0.5, 1, 0}$ . All lie in $[0, 1]$ ✓. Note $x_{B}$ and $x_{C}$ are exactly the non-overlapping regions where the true distributions share no support — precisely where the JS divergence saturates and the standard gradient dies (Ex 8 fixes this).

Case D — the vanishing signal, in numbers

The figure plots a saturated discriminator: an S-curve (sigmoid) that is nearly flat where the fakes live. The green tangent line is almost horizontal — its steepness is $\nabla_{x} D$ , and that flatness is the vanishing gradient.

Worked example

Ex 3 — Saturated critic kills the gradient (Cell D) A confident discriminator on a fake sample outputs $D (G (z)) = 0.01$ . The parent's gradient (original loss) is the product $scalar \frac{- 1}{1 - D ( G ( z ))} \cdot slope \nabla_{x} D (G (z)) \cdot knob-sensitivity \nabla_{θ} G (z) .$ Recall from the opening definitions: $\nabla_{x} D$ is the slope of the discriminator's S-curve at the sample (how much its score changes if the sample itself moves a hair), and $\nabla_{θ} G (z)$ says how the output picture moves when we wiggle each weight $θ$ — it is the only factor that actually touches the knobs we update. Suppose the saturated slope is $\nabla_{x} D = 0.002$ . Show the scalar is not the culprit.

Forecast: The parent warns the myth is " $\frac{- 1}{1 - D}$ shrinks". Does it?

Step 1 — the scalar: $\frac{- 1}{1 - 0.01} = \frac{- 1}{0.99} = - 1.0101$ . Why this step? We evaluate the supposed villain. It is $\approx - 1$ , an ordinary-sized number, not near zero.

Step 2 — the product magnitude (hold $\nabla_{θ} G$ fixed at $1$ so we isolate the other two factors): $(- 1.0101) \times 0.002 = - 0.00202$ Why? The whole signal is tiny because the slope $\nabla_{x} D = 0.002$ is tiny — the flat part of the S-curve. The scalar is innocent, and the knob-sensitivity $\nabla_{θ} G$ is just carried along as a fixed multiplier here.

Verify: $∣ - 0.00202∣ \approx 0.002$ , essentially the slope value. So the gradient magnitude tracks $\nabla_{x} D$ , not the scalar. Confirms the parent's correction: saturation (flat critic), not the prefactor, causes vanishing.

Case E — why the loss swap rescues training

Worked example

Ex 4 — Original vs non-saturating gradient scale (Cell E) Same confident critic, $D (G (z)) = 0.01$ . Compare the scalar prefactor of the two generator losses:

Original (minimize $lo g (1 - D)$ ): prefactor $\frac{- 1}{1 - D}$ .
Non-saturating (maximize $lo g D$ ): prefactor $\frac{1}{D}$ .

(Both losses share the same slope factor $\nabla_{x} D$ and the same knob-sensitivity $\nabla_{θ} G$ — only the scalar prefactor differs, so that is all we compare.)

Forecast: Which loss shouts louder when the generator is losing badly?

Step 1 — original prefactor: $\frac{- 1}{1 - 0.01} = - 1.0101$ . Why? As in Ex 3, roughly $- 1$ . Multiplying a tiny slope $\nabla_{x} D$ by $\approx 1$ leaves it tiny.

Step 2 — non-saturating prefactor: $\frac{1}{0.01} = 100$ . Why this step? When the fake is badly beaten ( $D \to 0$ ), $1/ D$ blows up. This amplifies the same tiny saturated slope $\nabla_{x} D$ , restoring a usable signal early in training.

Step 3 — the amplification ratio: $\frac{100}{1.0101} = 99.0$ . Why? This is how many times stronger the learning signal is at $D = 0.01$ . Nearly two orders of magnitude — the difference between "stuck" and "moving".

Verify: ratio $\approx 99$ . As $D (G (z)) \to 0$ , $1/ D \to \infty$ while $\frac{- 1}{1 - D} \to - 1$ ; the ratio diverges, matching the parent's claim that the losses "share fixed points but differ wildly in gradient magnitude when $D$ is confident". See optimization challenges.

Case F — the mode-collapse incentive

Worked example

Ex 5 — The penalty for trying variety (Cell F) The generator has a safe mode $x_{1}$ with $D (x_{1}) = 0.9$ . It considers a new, unpracticed mode $x_{2}$ with $D (x_{2}) = 0.3$ . Compute both generator losses $L_{G} = - lo g D$ and the gap.

Forecast: Which mode looks cheaper to the generator right now?

Step 1 — safe mode: $L_{G} (x_{1}) = - lo g (0.9) = 0.1054$ . Why? Low loss = "this fools the judge, keep doing it".

Step 2 — new mode: $L_{G} (x_{2}) = - lo g (0.3) = 1.2040$ . Why this step? The judge is suspicious of the never-seen mode, so producing it is punished. The generator's objective only cares about now, not about covering all modes.

Step 3 — the incentive gap: $1.2040 - 0.1054 = 1.0986$ . Why? This positive gap is the pull back toward the safe mode. Because it is positive, gradient descent shrinks variety — the mathematical seed of mode collapse.

Verify: gap $= ln (0.9/0.3) = ln 3 = 1.0986$ ✓. The gap equals $lo g \frac{D ( x _{1} )}{D ( x _{2} )}$ — it stays positive as long as the safe mode scores higher, which it always does until $D$ finally learns the new mode.

Case G — oscillation, no fixed point

The figure traces the "1 ↔ 7" cycle from the parent's MNIST example: generator output loops, discriminator focus lags one step behind, and the pair never lands on a fixed point.

Worked example

Ex 6 — A period-2 orbit (Cell G) Model a toy 2-mode loop. Let $s_{t} \in {1, 7}$ be which digit the generator makes at step $t$ , and the update rule be "switch to the digit the discriminator currently ignores". Start $s_{0} = 1$ ; the discriminator always chases the previous generator output. Track $s_{t}$ for 4 steps and ask: does it converge?

Forecast: If the judge always fights the last war, where does the generator go?

Step 1 — evolve the state: $s_{0} = 1 \to s_{1} = 7 \to s_{2} = 1 \to s_{3} = 7 \to s_{4} = 1$ . Why this step? Each step the generator flees to the mode the discriminator is not currently guarding, and the discriminator then chases it — one step late, forever.

Step 2 — detect the period: $s_{0} = s_{2} = s_{4} = 1$ and $s_{1} = s_{3} = 7$ . Period $= 2$ . Why? A repeating pair with no repeated consecutive state means a cycle, not a fixed point.

Step 3 — coverage check: distinct modes visited $= {1, 7}$ , i.e. $2$ of the $10$ MNIST digits. Why? Confirms the generator never covers the full distribution — $\frac{2}{10} = 20%$ of modes. Instability (Cell G) and collapse (Cell F) are the same disease seen from two angles.

Verify: period $2$ , coverage $2/10 = 0.2$ . The orbit has no fixed point (no $s_{t} = s_{t + 1}$ ), so the game does not converge — exactly the non-convergence the parent describes. Normalization tricks and progressive growing are practical dampers on such orbits.

Case H — the real-world value at optimum

Worked example

Ex 7 — What the objective equals at the perfect point (Cell H) A researcher claims: "at the global optimum, the GAN value $V^{*}$ is a fixed constant, no matter the data." Prove it numerically for the perfect generator, then check the disjoint-support extreme.

Forecast: If everything is balanced, is the answer $0$ ? Or something else?

Step 1 — value at $D^{*} = 0.5$ (perfect generator): from Ex 1, per-sample value $= 2 lo g (0.5) = - 1.3863$ . Why this step? At $p_{g} = p_{d a t a}$ the optimum value $V^{*}$ is a constant independent of the actual data shape — every sample contributes the same $- 1.3863$ .

Step 2 — identify the constant: $- lo g 4 = - 1.3863$ . Why this step? Plugging $JSD = 0$ (perfect match) into $V^{*} = - lo g 4 + 2 \cdot JSD$ gives exactly $V^{*} = - lo g 4$ . So the "fixed constant" the researcher claims is $- lo g 4$ — the theoretical floor of the game. Claim confirmed.

Step 3 — degenerate check, disjoint supports: if $p_{g}$ and $p_{d a t a}$ never overlap (Cells B & C everywhere), $JSD = lo g 2 = 0.6931$ , so $V^{*} = - lo g 4 + 2 lo g 2 = 0.$ Why this step? This is the worst case: the value jumps to $0$ and, crucially, $JSD$ is constant ( $lo g 2$ ) across all disjoint configurations → its gradient is $0$ → the generator gets no direction to move. This is the divergence-saturation that motivates WGAN (Ex 8).

Verify: $- lo g 4 = - 1.3863$ and $- lo g 4 + 2 lo g 2 = 0$ ✓. The value moves from $- 1.386$ (perfect) to $0$ (disjoint) — a bounded range, but the gradient dies at the disjoint end. That dead gradient is exactly the pathology the next example repairs.

Case I — the WGAN exam twist

Worked example

Ex 8 — Wasserstein stays informative where JS dies (Cell I) Exam-style: two point-mass distributions on the real line. $p_{d a t a}$ sits at position $0$ ; $p_{g}$ sits at position $d$ . They never overlap for $d \neq = 0$ . Compute (a) the JS divergence, (b) the Wasserstein-1 distance, for $d = 2$ and $d = 0.5$ . Which one gives a useful gradient?

Forecast: Two non-overlapping spikes — can JS tell $d = 2$ apart from $d = 0.5$ ?

Step 1 — JS divergence: for disjoint supports $JSD = lo g 2 = 0.6931$ regardless of $d$ . Why? Overlap is zero for any $d \neq = 0$ , so JS is stuck at its max (as in Ex 7). $JSD (d = 2) = JSD (d = 0.5) = 0.6931$ — identical. No gradient to shrink $d$ .

Step 2 — Wasserstein-1 (earth-mover): moving a unit mass a distance $d$ costs $W = ∣ d ∣$ . $W (d = 2) = 2, W (d = 0.5) = 0.5$ Why this step? $W$ is the cost to drag one histogram onto the other. It scales smoothly with $d$ , so $\frac{d W}{dd} = 1 \neq = 0$ : a clean "move left" signal everywhere.

Step 3 — a concrete 1-Lipschitz critic that achieves $W$ : take the critic $f (t) = - t$ . Its slope is $∣ f^{'} (t) ∣ = 1$ , so it just qualifies as 1-Lipschitz. The WGAN objective $E_{p_{d a t a}} [f] - E_{p_{g}} [f]$ evaluates on the two point masses to $f (0) - f (d) = 0 - (- d) = d$ . Why this step? WGAN defines the distance as the biggest value any 1-Lipschitz critic can produce. We exhibit one critic reaching $∣ d ∣$ , and no 1-Lipschitz critic can beat it (a slope- $1$ ramp is the steepest allowed), so the supremum equals $∣ d ∣$ — recovering the earth-mover number of Step 2 by an explicit construction rather than by assertion.

Verify: with $f (t) = - t$ : at $d = 2$ , $f (0) - f (2) = 0 - (- 2) = 2 = W$ ; at $d = 0.5$ , $0 - (- 0.5) = 0.5 = W$ ✓. Meanwhile $JSD = 0.6931$ for both $d$ (constant, useless gradient). This explicit critic is the whole reason WGAN cures the "disjoint-support" instability of Cells B/C/H.

Closing the drill hall

Every cell of the matrix is now worked with real numbers. The through-line: the same disease wears three faces. A too-confident discriminator saturates ( $\nabla_{x} D \to 0$ , Ex 3–4); disjoint distributions freeze the JS-divergence at $lo g 2$ so $V^{*}$ stops moving (Ex 7); and the generator, punished for variety (Ex 5), collapses onto a few safe modes and oscillates (Ex 6). WGAN (Ex 8) attacks all of them at the root by swapping the saturating probability $D$ for a smoothly-sloped critic. For where to go next: progressive growing and conditional control build stability into the architecture itself; normalization and optimization tactics tame the orbits in practice.

Recall Self-test — cover the answers

Perfect generator gives $D^{*} =$ ? ::: $0.5$ , and game value $V^{*} = - lo g 4 = - 1.386$ . What does $\nabla_{x} D$ mean? ::: the slope of the discriminator's S-curve with respect to the sample — how much the score changes if the sample moves a hair; it vanishes when $D$ saturates. A mode the generator never visits ( $p_{g} = 0$ ) gives $D^{*} =$ ? ::: $1$ (certainly real) — a collapse hole. Why does the gradient vanish when $D$ is confident? ::: The critic saturates so $\nabla_{x} D \to 0$ ; the scalar $- 1/ (1 - D) \to - 1$ is NOT the cause. Non-saturating prefactor at $D = 0.01$ ? ::: $1/0.01 = 100$ , vs original $\approx - 1$ (≈99× stronger). What is $L_{G}$ ? ::: the generator's per-sample loss $- lo g D (x)$ : small when $D$ is fooled, large when suspicious. Mode-collapse gap for $D (x_{1}) = 0.9, D (x_{2}) = 0.3$ ? ::: $ln 3 = 1.099$ — penalty for variety. $V^{*}$ at disjoint supports? ::: $- lo g 4 + 2 lo g 2 = 0$ , and JSD frozen at $lo g 2$ so its gradient dies. Why WGAN? ::: JS is stuck at $lo g 2$ for disjoint supports (no gradient); $W = ∣ d ∣$ varies smoothly (gradient 1), realised by the 1-Lipschitz critic $f (t) = - t$ .