Exercises — GAN training instability and mode collapse

This is mode collapse. It is stable because the generator's objective only rewards fooling the discriminator right now; producing one "safe" output that reliably fools $D$ gives low loss, and there is no term in $L_G$ that rewards variety across the distribution.

• Student A → training instability (oscillation / non-convergence).
• Student B → mode collapse (limited variety).

False. The original loss minimizes $\log (1-D(G(z)))$. The non-saturating fix has the generator maximize $\log D(G(z))$ instead — same fixed points, stronger early gradients.

$D^{\ast }(x_0)=\frac{0.6}{0.6+0.2}=\frac{0.6}{0.8}=\mathrm{0.75.}$ The discriminator is 75% sure this point is real — sensible, since real density is 3× the fake density here.

$L_G(x_1)=-\ln (0.9)\approx 0.1054,L_G(x_2)=-\ln (0.3)\approx \mathrm{1.2040.}$ The new-mode loss is larger by a factor $\approx 1.2040/0.1054\approx 11.4$. The generator is ~11× more penalized for exploring $x_2$ — this is the numerical heart of mode collapse.

$D^{\ast }(x)=\frac{p_{data}}{p_{data}+p_{data}}=\frac{p_{data}}{2p_{data}}=\frac{1}{2}.$ The best possible discriminator is reduced to a coin flip — it cannot distinguish real from fake, which is exactly the equilibrium we want.

The student is wrong: at $D(G(z))\approx 0$ the scalar $\frac{-1}{1-0}=-1$ is a constant of magnitude 1, not small. Look at the blue sigmoid curve in the figure: a confident discriminator lives on the flat tails near output $0$ or $1$. Flat means slope $\approx 0$, i.e. ${\nabla }_{x}D\to 0$. The pink arrows show the slope shrinking to nothing on the tail. So the product dies because of the saturated input-gradient, not the scalar. The non-saturating loss swaps the scalar for $\frac{1}{D(G(z))}$, which blows up as $D\to 0$ and rescues the signal.

$L_D=-{\mathbb{E}}_{x\sim p_{data}}[\log D(x)]-{\mathbb{E}}_z[\log (1-D(G(z)))]$. The second term only contains samples $G$ actually produces now. If $G$ stops making "1"s and makes "7"s, the gradient pushing $D$ to catch "1"s disappears; $D$'s decision boundary drifts toward catching "7"s and un-learns "1". When $G$ swings back to "1", it faces a $D$ that has forgotten it — low loss again. There is no memory term, so the cycle self-perpetuates.

At $D=0.001$: $\frac{1}{0.001}=1000$. At $D=0.5$: $\frac{1}{0.5}=2$. Early in training $D$ is confident about fakes, so $D(G(z))$ is tiny and the prefactor is huge. This large number amplifies the small saturated ${\nabla }_{x}D$, restoring a usable learning signal exactly when the original loss had none. As $G$ improves ($D\to 0.5$) the prefactor calmly relaxes to $2$. So the "explosion" is self-limiting and targeted.

(a) Vanishing gradients: the critic has no sigmoid, so it never saturates onto a flat tail; ${\nabla }_x{f}_w$ stays non-zero everywhere. The signal that the saturated ${\nabla }_{x}D$ destroyed can no longer die. (b) Mode collapse: the Wasserstein distance $W(p_{data},p_g)={\sup }_{\Vert f{\Vert }_L\le 1}{\mathbb{E}}_{p_{data}}[f]-{\mathbb{E}}_{p_g}[f]$ measures the cost of transporting mass across all of space. A missing mode leaves probability mass un-transported, which the sup detects and penalizes — so the gradient pulls toward every mode, not just the visible ones. Synthesis: (a) is a property of the critic's output shape; (b) is a property of the distance being optimized. WGAN changes both at once.

Maximum difference $=1\cdot 2=2$. The Lipschitz cap limits how steep the critic can be, so ${\nabla }_x{f}_w$ can never blow up to arbitrarily large values. This keeps the generator's gradient bounded and well-conditioned — the objective now correlates smoothly with sample quality instead of spiking. Enforced by weight clipping $w\leftarrow \text{clip}(w,-c,c)$ or a gradient penalty.

From 2.2, exploring a fresh mode costs ~11× the loss of repeating a safe one, so $G$ greedily collapses onto the safe mode. From 3.2, $L_D$ only trains against modes $G$ currently produces, so $D$ specializes on the safe mode and its boundary drifts, forgetting others. Once $D$ over-specializes, a different safe mode becomes cheap for $G$, so $G$ jumps to it; $D$ chases, forgetting the first; when $G$ returns, the first mode is cheap again. The asymmetric incentive plus the memoryless $L_D$ produce a limit cycle — the 1→7→1 oscillation.

Differentiate: $g^{\prime }(d)=\frac{a}{d}-\frac{b}{1-d}$. Set to zero: $\frac{a}{d}=\frac{b}{1-d}\Rightarrow a(1-d)=bd\Rightarrow a=d(a+b)\Rightarrow d^{\ast }=\frac{a}{a+b}.$ Substituting back, $D^{\ast }(x)=\frac{p_{data}(x)}{p_{data}(x)+p_g(x)}$. Second derivative $g^{\prime \prime }(d)=-\frac{a}{d^2}-\frac{b}{(1-d{)}^2}<0$, confirming a maximum. This is exactly the parent's result, now derived from scratch.

With $p_{data}=p_g$, $D^{\ast }(x)=\frac{1}{2}$ everywhere, so $V={\mathbb{E}}_{p_{data}}[\log \frac{1}{2}]+{\mathbb{E}}_{p_g}[\log (1-\frac{1}{2})]=\log \frac{1}{2}+\log \frac{1}{2}=2\log \frac{1}{2}=-2\log 2=-\log 4.$ Numerically $-\log 4\approx -1.386$. Interpretation: at the global optimum the value is a fixed constant $-\log 4$; the generator cannot push it lower because it has already matched $p_{data}$ perfectly. This is the theoretical "convergence target."

Intervention 1 — Non-saturating loss (maximize $\log D(G(z))$). Mechanism repaired: the saturated input-gradient ${\nabla }_{x}D\to 0$ from symptom (i). The $\frac{1}{D}$ prefactor amplifies the tiny gradient. Predicted change: $G$'s loss starts moving again; samples un-freeze. Intervention 2 — Switch to WGAN critic + Lipschitz constraint (see 4.5.07-Wasserstein-GAN-and-improved-training). Mechanism repaired: (a) removes sigmoid saturation entirely; (b) the Wasserstein distance pulls toward all modes, fixing the 2-of-10 collapse in symptom (ii). Predicted change: sample variety climbs toward all 10 digits; loss now tracks quality. (Optional third: balance update ratio / lower $D$'s learning rate — see 3.4.05-Optimization-challenges-in-deep-learning — to stop $D$ winning in 50 steps. Also 4.3.06-Batch-normalization-and-alternatives can stabilize the training dynamics.)

Which term is a constant $-1$, not small, when $D(G(z))\to 0$? ::: the original-loss scalar $\frac{-1}{1-D(G(z))}$. What actually vanishes to kill the gradient? ::: the input-gradient ${\nabla }_{x}D$, because a confident $D$ saturates onto a flat tail. Global game value at optimum (natural log)? ::: $-\log 4\approx -1.386$. Ratio of new-mode loss to safe-mode loss in Ex 2.2? ::: about $11.4$.