Generator and discriminator dynamics: Question bank

True or false — justify

The optimal discriminator

D^{*} (x) = \frac{1}{2}

everywhere only when the generator has perfectly matched the data.

True.

D^{*} (x) = \frac{p _{d a t a} ( x )}{p _{d a t a} ( x ) + p _{g} ( x )}

equals

\frac{1}{2}

precisely when

p_{g} (x) = p_{d a t a} (x)

at every

x

; anywhere the two densities differ,

D^{*}

tilts away from

0.5

If the discriminator loss stops decreasing, the GAN has converged.

False. GAN training is a two-player game, not one loss to minimise; a flat D-loss can mean D is momentarily beaten, or that both players are stuck oscillating around a saddle — not a stable Nash equilibrium.

At the ideal equilibrium the generator's loss goes to zero.

False. When

D (G (z)) = 0.5

, the non-saturating loss is

- lo g (0.5) \approx 0.69

, not zero. A "flatlining near 0.69" for both players is the healthy sign, not a zero loss.

The generator can see real data directly during training.

False. G only ever receives gradient signal through D; it never touches

p_{d a t a}

samples. All its information about "what real looks like" is second-hand, filtered by the discriminator.

Maximising

V (D, G)

over

D

is the same task as ordinary supervised binary classification.

True. Maximising

E [lo g D (x)] + E [lo g (1 - D (G (z)))]

is exactly minimising binary cross-entropy with label 1 on real and 0 on fake — the sign flip is the only difference.

The first term

E_{x \sim p_{d a t a}} [lo g D (x)]

matters when computing G's gradient.

False. That term has no dependence on G's parameters

θ_{g}

, so its gradient w.r.t. G is zero; G effectively only sees the fake-sample term.

The non-saturating loss

- lo g D (G (z))

changes where the Nash equilibrium sits.

False. It changes the gradient landscape (stronger signal when G is failing), but the fixed point is still

p_{g} = p_{d a t a}

D^{*} = \frac{1}{2}

. It's a re-parameterisation for optimisation, not a new game.

Spot the error

"Since G minimises

lo g (1 - D (G (z)))

, we should also let G push the

lo g D (x)

term down."

Error: G has no control over

lo g D (x)

— that term depends only on real samples and D. Trying to "push it down" is meaningless; it's a constant from G's viewpoint.

"D should be trained to output exactly 1 for real and 0 for fake, so we always want maximally confident D."

Error: an over-confident D saturates (

D (G (z)) \to 0

), killing G's gradient in the original loss. We want D good but not saturated, which is why we alternate updates instead of training D to convergence each round.

"The minimax objective always trains stably because it has a unique global optimum."

Error: existence of an optimum doesn't guarantee gradient descent-ascent reaches it. The dynamics can cycle around the saddle indefinitely; convergence to the equilibrium is not guaranteed by having one.

"If

D (G (z)) = 0.05

, the original loss

lo g (1 - D (G (z)))

gives a big gradient because D is very wrong about... no, right about the fake."

Error: at

D (G (z)) = 0.05

D is correctly confident it's fake, and

lo g (1 - 0.05) \approx 0

with a tiny slope — the vanishing gradient case. The original loss gives a small gradient exactly here, which is why the non-saturating form exists.

"Mode collapse means the discriminator collapsed and outputs a constant."

Error: mode collapse is a generator failure — G maps many noise vectors to one (or few) outputs, ignoring the diversity of real data. See 4.5.08-Mode-collapse-in-GANs; it is not D outputting a constant.

"We ascend the gradient for both D and G because both are being maximised."

Error: D ascends

V

(maximiser), but G descends

V

(minimiser). Only in the non-saturating reformulation do we phrase G's step as maximising

lo g D (G (z))

— and even then it's a different objective, not the same one.

"Because

D^{*} (x)

is a probability, it must lie strictly between 0 and 1."

Error: at the boundary where

p_{d a t a} (x) = 0

but

p_{g} (x) > 0

D^{*} (x) = 0

exactly; where

p_{g} (x) = 0

but

p_{d a t a} (x) > 0

D^{*} (x) = 1

. The optimum does hit the endpoints in degenerate regions.

Why questions

Why do we use

lo g

inside the objective rather than raw probabilities?

Log turns the product of per-sample likelihoods into a sum (stable, differentiable), and it is the natural object from maximum-likelihood / cross-entropy — matching D's task of classifying real vs fake.

Why does the original generator loss vanish early in training?

Early on G is bad, so D confidently outputs

D (G (z)) \approx 0

; then

lo g (1 - D (G (z))) \approx lo g 1 = 0

and its slope is nearly flat, so G receives almost no gradient exactly when it needs to learn most.

Why does the non-saturating loss

- lo g D (G (z))

fix this?

Its gradient scales like

\frac{1}{D ( G ( z ))}

, which blows up when

D (G (z))

is tiny — giving G a strong learning signal precisely when it is failing, instead of a vanishing one.

Why do we typically take

k

steps on D per single step on G?

To keep D close to its optimum

D^{*}

for the current G, so that G's gradient points in a meaningful direction; if D lags too far behind, G's signal becomes misleading.

Why is

D^{*} (x) = \frac{p _{d a t a}}{p _{d a t a} + p _{g}}

called Bayes-optimal?

It's the posterior probability that a sample came from the real class assuming equal priors on real/fake — the best possible classifier given the two densities, so no discriminator can do better against that fixed G.

Why can't the discriminator "win" permanently even if it's much stronger than G?

Because if D wins hard, G's gradient vanishes (original loss) or D provides an easy target to climb (non-saturating loss); the game structure forces D to keep re-adjusting as G shifts

p_{g}

, so no permanent win exists at equilibrium.

Why does Wasserstein GAN change the loss instead of just tuning learning rates?

The JS-divergence implicit in the original objective saturates (flat gradients) when

p_{g}

and

p_{d a t a}

barely overlap; WGAN replaces it with a distance whose gradient stays informative even for disjoint supports — a structural fix, not a hyperparameter one.

Edge cases

What is

D^{*} (x)

at a point where

p_{d a t a} (x) = p_{g} (x) > 0

Exactly

0.5

— D is genuinely undecided there because a sample is equally likely to come from either source. This is the local signature of a matched region.

What happens to

D^{*} (x)

where the real density is zero but G still generates samples (

p_{g} (x) > 0

D^{*} (x) = \frac{0}{0 + p _{g}} = 0

; D can be perfectly certain such outputs are fake, which is why "hallucinated" off-manifold generator samples are so easy to catch.

What is G's ideal

p_{g}

, and why is

D^{*} = \frac{1}{2}

everywhere at that point?

Ideal is

p_{g} = p_{d a t a}

at every

x

; then the ratio

\frac{p _{d a t a}}{p _{d a t a} + p _{g}} = \frac{1}{2}

uniformly, meaning D has literally no information to separate the two distributions.

In the numerical case

D (G (z)) = 0.5

, what does the non-saturating generator loss evaluate to, and is that good?

- lo g (0.5) \approx 0.69

; this is the healthy equilibrium value — G isn't "losing", it has driven D to maximal uncertainty, which is the goal.

If G suffers mode collapse but still fools D on its single mode, has it reached the true equilibrium?

No. Fooling D on one mode gives low loss on expectation over that mode, but

p_{g} \neq = p_{d a t a}

globally, so it is a degenerate fixed point of the alternating updates, not the true Nash equilibrium.

What happens to the gradient descent-ascent dynamics if D and G have exactly matched, opposing gradients each step?

They can enter a stable cycle (limit-cycling) around the equilibrium rather than converging — a hallmark of the non-convex, non-concave nature of the min-max game and a known source of training instability.

Recall Fastest self-check

Optimal discriminator formula ::: $D^{*} (x) = \frac{p _{d a t a} ( x )}{p _{d a t a} ( x ) + p _{g} ( x )}$ Value of $D^{*}$ at perfect equilibrium ::: $\frac{1}{2}$ everywhere, because $p_{g} = p_{d a t a}$ . Why non-saturating loss exists ::: Original $lo g (1 - D (G (z)))$ has vanishing gradient when D is confident; $- lo g D (G (z))$ gives a strong gradient there. Whose failure is mode collapse ::: The generator's — it ignores diversity and maps to few outputs.