4.5.8 · D5Generative Models
Question bank — GAN training instability and mode collapse
This is a misconception hunt for GAN training instability and mode collapse. Every line below is a Question ::: Answer reveal — read the question, commit to an answer in your head, THEN reveal. If your gut answer differs from the reasoning, you just found a trap worth remembering.
Before we start, three plain-word anchors so nothing here uses an unearned term:
Related builds if a line surprises you: 4.5.01-Generative-Adversarial-Networks-fundamentals, 4.5.07-Wasserstein-GAN-and-improved-training, 3.4.05-Optimization-challenges-in-deep-learning, 4.3.06-Batch-normalization-and-alternatives.
True or false — justify
A perfect generator makes the discriminator useless.
True — if everywhere, the optimal judge is for every , i.e. pure coin-flipping. It literally cannot do better than guessing.
The vanishing-gradient problem happens because the prefactor shrinks to zero.
False — that prefactor approaches the constant , not . The gradient dies because a saturated has , and that factor multiplies everything to nothing.
The non-saturating loss () changes where the game's fixed points are.
False — it shares the exact same optimum (). It only changes the gradient magnitude early on, when is confident and needs a loud signal.
Mode collapse means the generator's outputs are low quality.
False — collapsed outputs are often high quality and fool well; they just lack variety. Quality and diversity are separate failures.
If training loss is oscillating wildly, the model is definitely diverging.
False — a minimax game is a moving target, so bounded oscillation is normal and can coexist with improving samples. GAN loss curves are poor progress meters (this is a motivation for the WGAN critic value, which does track quality).
A discriminator that is "too good" is always desirable because it classifies accurately.
False — a too-confident saturates, sends , and starves of gradient. You want strong enough to guide but not so strong it stops teaching.
WGAN's critic outputs a probability just like a standard discriminator.
False — the critic outputs an unbounded real number (no sigmoid). That's the whole point: no sigmoid means no saturation ceiling to flatten out on.
Mode collapse and vanishing gradients are the same phenomenon.
False — vanishing gradients is a no-signal failure (nothing learns); mode collapse is a wrong-incentive failure (learning happens, but toward a shortcut). They can even occur together.
The Lipschitz constraint in WGAN is optional decoration.
False — the Wasserstein distance formula only equals the sup over 1-Lipschitz functions. Drop the constraint and can blow up to , making the objective meaningless. Clipping or gradient penalty enforces it.
Spot the error
"D's gradient vanishes, so we should train D more to fix it."
Backwards — over-training is what caused the saturation. The cure is to weaken/regularize or switch to the non-saturating loss, not to strengthen .
"Since gives huge in the non-saturating loss, the gradient is guaranteed large."
The large prefactor helps but doesn't guarantee — it multiplies a small . It amplifies a fading signal; if is truly flat, even times near-zero can stay small. It's a rescue, not immunity.
"Mode collapse is impossible if the generator has enough capacity."
Capacity is irrelevant — collapse is an incentive problem from the game dynamics, not a representation problem. A huge network still collapses if fooling with a few modes is the easy win.
"The discriminator remembers every mode it has ever punished."
No — only updates on the samples it currently sees. If stops producing a mode, forgets to check for it (drift), which lets cycle back to that mode later.
"We can read off generator progress directly from the standard GAN loss going down."
The two losses are adversarial, so 's loss dropping often just means temporarily lost, not that samples improved. Standard GAN loss does not correlate with sample quality (WGAN's critic value is a fix for exactly this).
"Using vs is just an algebra rearrangement with no practical difference."
They give different gradients when is confident: the original vanishes, the non-saturating one amplifies. Same fixed point, wildly different early-training behavior.
Why questions
Why does the generator get penalized for trying a fresh, genuine mode ?
Because hasn't seen produce -type fakes, so it stays suspicious ( low), giving a high loss . Honesty is punished short-term; the safe old mode has lower loss.
Why does the Wasserstein distance help cover all modes, not just visible ones?
The critic assigns a smooth real-valued "elevation" everywhere, so its gradient pulls toward across the whole space — including regions currently ignores. A saturating sigmoid gives zero pull there instead.
Why do we run several critic updates per generator update in WGAN?
The Wasserstein estimate is only meaningful when is near-optimal (a good 1-Lipschitz "measuring stick"). Under-trained critic = wrong distance = misleading generator gradient, so we sharpen the critic first.
Why does the non-saturating loss give a strong signal precisely when the original gives a weak one?
Both are triggered by . The original then has flat gradient (dead); the non-saturating one has prefactor , so the same confident- regime that kills one loss energizes the other.
Why can't we just add a "produce variety" term and be done with mode collapse?
You can nudge it (minibatch discrimination, unrolled GANs, WGAN), but the root cause is that 's objective only rewards fooling now. Any fix must change the incentive so that missing modes actually costs the generator.
Edge cases
What is in a region where but (generator hallucinates)?
— the judge correctly flags "pure fake territory". Here should feel maximum pressure to retreat from that region.
What is where but (a missed mode)?
— real data lives here that never visits. Under a standard sigmoid this is a saturated , so gets no gradient telling it to cover the mode — exactly why standard GANs abandon modes.
If and have no overlap at all (disjoint supports), what happens to a standard discriminator?
It achieves perfect separation, saturates fully ( on real, on fake), and everywhere useful — total gradient death. This disjoint-support case is the headline argument for switching to the Wasserstein distance, which still gives a finite, informative value.
What does mode collapse look like at the extreme limit?
maps every noise to essentially one output — the whole rich distribution shrinks to a single point. Diversity is zero even though that one sample may perfectly fool .
At the ideal equilibrium , what gradient does receive?
Zero net gradient — everywhere gives no direction to move, which is correct because there's nothing left to improve. The danger is telling this good zero-gradient apart from the bad saturated zero-gradient of a mode-collapsed state.
If you clip WGAN weights too aggressively (tiny ), what breaks?
The critic loses capacity and collapses toward simple functions, so the Wasserstein estimate degrades and gradients weaken — a reason gradient penalty is often preferred over hard clipping (see 4.5.07-Wasserstein-GAN-and-improved-training).
Recall Fastest self-test
Vanishing gradient is caused by ::: a saturated discriminator making — NOT the prefactor. Mode collapse is fundamentally a ::: bad-incentive problem: is rewarded for fooling now, not for covering all modes. WGAN's core structural change is ::: replacing the sigmoid probability judge with an unbounded 1-Lipschitz critic, giving non-vanishing gradients even on disjoint supports.