4.5.6 · D5Generative Models
Question bank — Generative Adversarial Networks (GAN) framework
This page is a stress-test for your understanding of the Generative Adversarial Networks (GAN) framework. Each line is a Question ::: Answer reveal — cover the right side, commit to an answer with a reason, then check. If your reason was wrong even when the verdict was right, you found a trap.
Reminders of the notation you will need (all defined in the parent):
- — the Generator, turning noise into a fake sample.
- — the Discriminator, outputting the probability that is real, a number in .
- — the noise distribution: a fixed, easy-to-sample source of randomness (e.g. a standard Gaussian or a uniform box). It is chosen by us, never learned; reshapes this simple noise into the complex data distribution.
- — the distribution of real data; — the distribution of the Generator's outputs (what you get by pushing through ).
- — the value function both networks fight over.
Two more objects show up repeatedly below. Rather than defer them, here they are in plain words:
The three figures below are your visual anchors; the questions refer to them by number.
Figure 1 — Verdicts converging: how 's outputs on real and fake data both drift to .
Figure 2 — Saturating vs non-saturating loss: why the naive Generator loss goes flat and the fix stays steep.
Figure 3 — The JSD landscape: the divergence the ideal Generator minimizes, and its flat no-gradient region.
True or false — justify
At equilibrium the Discriminator has "lost", so it is a failed network.
False. Outputting everywhere means : the two distributions are genuinely identical, so no classifier could beat a coin flip. hasn't failed — it has reached the point of maximum uncertainty, which is precisely the sign that succeeded.
A higher for a fake image always means the Generator is winning.
Roughly true at that instant, but misleading: it only says was fooled on that sample. What matters is the trend across training, since is simultaneously being retrained to correct such mistakes.
The Generator ever sees a real data sample during training.
False. only ever receives noise and gradient signals routed back through . It never touches directly — it learns the data distribution entirely secondhand via the Discriminator.
Minimizing and maximizing push in the same direction for .
True — both drive upward toward (fully fooled), so moves the same way under either. They differ only in gradient strength when is small (see Figure 2), which is why the non-saturating form is preferred early in training.
Because the game is , the Generator is trying to make as negative as possible.
False. only controls the second term (); the first term does not depend on at all, so cannot touch it. just drives up.
The Discriminator's optimal output is only after training finishes.
False. That formula is the optimal for whatever currently is, at any moment. Training tries to keep near this optimum for the present Generator, which is why we take discriminator steps.
If for every input, the GAN must have converged.
False. also happens at random initialization when hasn't learned anything. Convergence requires because , not because is untrained — the generated images tell you which case you are in.
GANs explicitly write down and maximize a likelihood of the data, like classic maximum-likelihood models.
False. GANs never evaluate or a likelihood; the "realistic" criterion is learned implicitly by . This is the whole point contrasted in Discriminative vs Generative Models.
The KL divergence appears anywhere in the standard GAN objective.
True but subtle — at the optimal , the value function reduces to a Jensen–Shannon divergence, which is itself built from two KL terms (see the definition callout above). Plain forward-KL is not what a GAN minimizes.
Spot the error
"We train and jointly with one shared gradient descent step to keep it simple."
Error: their objectives conflict, so a single joint step gives each a corrupted signal. The parent uses alternating updates so each network adapts to the other's current fixed state. See Gradient Descent Optimization.
"During the Generator update we backprop through and update both and ."
Error: we backprop through (to get the chain-rule gradient ) but only is updated. Updating here would let sabotage its own signal.
"We minimize for — it works fine even when is terrible."
Error: when is terrible, , making with a nearly flat slope — the vanishing gradient problem (the flat magenta curve in Figure 2). The non-saturating fixes exactly this.
" outputs the class label of the image (which digit it is)."
Error: outputs a single scalar in — the probability that is real vs fake, not a class. Labelling what the image contains is the job of a conditional setup like Conditional GANs (cGAN).
"Since uses log-probabilities, the Discriminator loss is just plain mean-squared error."
Error: the objective is a binary log-loss — it is Cross-Entropy Loss over the real/fake labels. MSE is a different (WGAN-adjacent or least-squares GAN) design choice, not the standard formulation.
"If the Generator collapses to producing one perfect image, that's ideal — it fools ."
Error: that's mode collapse. Fooling on one mode ignores the diversity of ; the goal is across all modes. Fixes live in Mode Collapse Solutions.
"We take Generator steps per Discriminator step to keep ahead."
Error: it's the reverse — Discriminator steps per one Generator step, so stays near optimal and hands a meaningful gradient.
Why questions
Why do we use in the objective rather than raw probabilities ?
Three reasons, but the key one is gradient shaping: the slope of is , which grows as (see the steep violet curve in Figure 2) — so a confidently-wrong verdict produces a large correcting gradient. (Logs also turn products of independent-sample probabilities into sums, and improve numerical stability.)
Why must the Generator's gradient travel through the Discriminator?
Because has no direct feedback from real data; its only measure of "realism" is 's verdict. The chain rule literally requires differentiating 's forward pass w.r.t. its input.
Why can't we just directly solve the minimax problem in closed form?
and are deep non-convex networks over millions of parameters; there's no analytic saddle point. We approximate it with iterative alternating gradient steps, hoping to approach a Nash Equilibrium.
Why do the numbers and both drift toward as training succeeds?
As , real and fake become statistically identical, so the best possible classifier is forced to guess — its output collapses to the maximum-entropy value for both inputs (the converging curves in Figure 1).
Why does collapse to the constant at equilibrium?
At equilibrium for every , so each expectation's integrand becomes the same constant regardless of the sample. The expectation of a constant is just that constant, so .
Why is a GAN called adversarial rather than cooperative?
The two networks optimize opposing terms of the same : every gain for is a loss for and vice versa. The competition — not cooperation — is what ratchets both toward excellence.
Why do we sample fresh noise for the Generator update instead of reusing the noise from the Discriminator step?
Fresh gives an unbiased Monte-Carlo estimate of for the Generator's own objective and avoids overfitting to the specific fakes just judged.
Why is a GAN called adversarial rather than cooperative? (edge form)
Skip — see above.
Why does a poorly-trained Discriminator hurt the Generator?
A weak gives near-random verdicts, so the gradient it passes to points in no useful direction — learns nothing about how to look more real.
Edge cases
What is at the very start of training (random init)?
About — but for the wrong reason: is untrained and guessing. This is why raw output values alone can't tell "start" from "convergence"; you must also inspect the generated images (contrast the endpoints in Figure 1).
What happens to if ever becomes perfect and outputs exactly for fakes?
— the loss explodes. In practice outputs are clamped away from and , and this pathology is one motivation for the smoother Wasserstein GAN (WGAN) loss.
What if and have no overlap in their supports?
The Jensen–Shannon divergence saturates to its cap of (a flat region, Figure 3), so gradients vanish and gets no signal to move — exactly the failure the earth-mover distance in Wasserstein GAN (WGAN) was designed to escape.
Degenerate case: the Generator ignores its noise input entirely.
Then produces the same output regardless of — total mode collapse to a single point. becomes a spike, unable to match a spread-out .
How would you detect mode collapse during training?
Watch for the Generator's samples losing variety (many near-duplicate outputs from different ), the Generator loss oscillating while 's loss drops, or a nearest-neighbour diversity metric flatlining. Sample-grid inspection each epoch is the cheapest check.
How do you remediate mode collapse in practice?
Give a view of batch statistics so it can penalize low diversity (minibatch discrimination), unroll a few steps so anticipates 's reaction (unrolled GANs), or switch to the Wasserstein GAN (WGAN) loss — all beyond the vanilla objective; see Mode Collapse Solutions.
Edge case: the two learning rates for and are badly mismatched.
If learns far faster it becomes near-perfect, sending into the flat saturating region (no gradient, Figure 2); if outpaces , never provides a sharp target. Balanced rates (or two-timescale updates) keep the arms race productive.
Edge case: training oscillates or diverges instead of settling.
The minimax objective has no descent guarantee — the pair can cycle around the equilibrium (like two players endlessly best-responding) rather than converge. Damping tricks (lower learning rates, gradient penalties, historical averaging) are used to stabilize it.
Edge case: "discriminator forgetting."
As shifts, may over-adapt to 's current fakes and forget how to reject earlier failure modes, letting cycle back to old bad samples. Replaying past generated samples in 's batches counteracts this.
Edge case: the real dataset is extremely small.
can memorize and perfectly classify the few real points, driving toward flatness; overfits to those memorized samples rather than the true distribution — a reason practitioners pair GANs with Data Augmentation with GANs.
Recall One-line self-check
Does everywhere prove convergence? No — it also occurs at random initialization; you must confirm via the sample quality.