DCGAN, WGAN, StyleGAN: Question bank — AI-ML deep dive

This page is a misconception hunt for the parent topic. Each line below is a Question ::: Answer reveal — read the question, commit to an answer out loud, then reveal. The answers are short but they always give the reason, because in GANs the reason is the whole game.

Before we start, three plain-word anchors so nothing below uses a term you have not met:

True or false — justify

TF1. "DCGAN changed the GAN loss function to make training stable."

False. DCGAN kept the same minimax loss; it only changed the architecture (convolutions, batch norm, activations). The loss stayed "shaky" — WGAN is the one that fixed the loss.

TF2. "Batch normalization should be applied to every layer of both networks."

False. It is skipped at the discriminator input (real/fake images should be judged per-sample, not normalized across a mixed batch) and at the generator output (tanh needs the full range to produce true colors).

TF3. "Jensen–Shannon divergence goes to infinity when real and fake distributions don't overlap."

False. It saturates at the constant $lo g 2$ . A constant has zero gradient — that flatness, not infinity, is what starves the generator.

TF4. "Wasserstein distance is undefined when the two distributions have disjoint supports."

False. It stays finite and varies smoothly: it simply reports the average distance mass must travel, so it still points a direction even when nothing overlaps.

TF5. "In WGAN the critic outputs a probability between 0 and 1, just like a discriminator."

False. The critic outputs an unbounded real score (an estimate of

E [f (x)]

), not a probability. There is no sigmoid at the end.

TF6. "Weight clipping in WGAN exactly enforces the 1-Lipschitz constraint."

False. It only approximately enforces it and is admittedly crude — it biases weights toward the clip boundary

\pm c

and wastes model capacity.

TF7. "The generator's Adam should use the default

β_{1} = 0.9

for best DCGAN results."

False. DCGAN uses

β_{1} = 0.5

; the high default momentum causes oscillation in the adversarial back-and-forth.

TF8. "A larger Wasserstein loss value always means worse generated images."

True(-ish), and that's the point. Because

W

tracks real distance, the critic's loss correlates with sample quality — a property JS-based GANs famously lack. Lower critic estimate → closer distributions → better images.

TF9. "Transposed (fractional-strided) convolutions and pooling do the same job in the generator."

False. Pooling is fixed downsampling; the generator needs learnable upsampling, which transposed convolutions provide by doubling spatial size each step.

Spot the error

SE1. "We use stride=1 transposed convolutions to add capacity while growing the image from 4×4 to 8×8."

Error: stride=1 does not change spatial size, so the map stays 4×4. Doubling requires stride=2. Extra capacity without upsampling never reaches 64×64.

SE2. "We put BatchNorm right after the discriminator's input layer so the real images are normalized."

Error: that layer sees a mixed batch of real and fake, so BatchNorm leaks batch statistics between them and prevents per-sample judgment. Leave the input layer un-normalized.

SE3. "WGAN removed the sigmoid, so we still call it a 'discriminator' scoring true/false."

Error: with no sigmoid the network no longer classifies — it scores. That's why the paper renames it the critic; treating its output as a probability breaks the Kantorovich–Rubinstein interpretation.

SE4. "To enforce Lipschitz-1 we clip the activations to

[- 0.01, 0.01]

Error: WGAN clips the weights

w

, not activations. Bounded weights bound the gradient (by the chain rule), which bounds how fast

f

can change — that's the Lipschitz link.

SE5. "The generator's final layer uses ReLU so outputs are non-negative pixel values."

Error: the final layer uses tanh, mapping to

[- 1, 1]

to match images normalized to that range. ReLU would clip away all the negatives and destroy dark tones.

SE6. "Since the critic maximizes

E [f (x_{r})] - E [f (x_{g})]

, we should train it only once per generator step, like a normal GAN."

Error: WGAN trains the critic $n_{critic} \approx 5$ times per generator step, because a well-trained critic gives a better Wasserstein estimate — and a better estimate is safe here (unlike the vanishing-gradient danger of an over-trained standard discriminator).

SE7. "DCGAN uses fully connected layers between conv blocks to mix global information."

Error: DCGAN removes FC hidden layers; they destroy spatial locality. Only the very first projection of

z

into a 4×4 seed is FC-like.

Why questions

WHY1. Why does a constant JS divergence (

lo g 2

) hurt the generator specifically?

The generator learns by following the gradient of the loss. A constant has gradient zero everywhere, so it delivers a "you're wrong" verdict with no direction to improve — a compass with no needle.

WHY2. Why is the Wasserstein distance nicknamed Earth Mover's Distance?

Picture

P_{r}

and

P_{g}

as two piles of dirt;

W

is the minimum total work (mass × distance) to reshape one pile into the other. It answers "how far must each grain move?", which is a smooth, directional quantity.

WHY3. Why does the 1-Lipschitz constraint keep the critic's loss meaningful?

Without it the critic could scale its weights up arbitrarily, blowing the loss to infinity while learning nothing real. Capping the slope forces

f

to encode actual distances between distributions, which is exactly what

W

needs.

WHY4. Why do transposed convolutions with 4×4 kernel and stride 2 avoid checkerboard artifacts better than some other choices?

A kernel size that is an even multiple of the stride lets each output pixel receive an even, overlapping contribution, so the upsampling covers the grid uniformly instead of leaving periodic bright/dark squares.

WHY5. Why LeakyReLU in the discriminator but plain ReLU in the generator?

LeakyReLU keeps a small gradient for negative inputs so discriminator neurons don't "die" (stop learning), which matters when its gradients feed back into the generator. The generator can afford ReLU's sparsity because it's building, not judging.

WHY6. Why does DCGAN halve channels while doubling spatial size each layer?

It moves from "few big-picture features on a tiny grid" to "many fine details on a large grid" — coarse structure first, texture last — while keeping the compute per layer roughly balanced.

WHY7. Why does WGAN's critic loss correlate with image quality when standard GAN loss does not?

Because

W

is a genuine metric of distance between distributions, its estimate shrinks smoothly as the fake distribution approaches the real one. JS's saturation makes the standard loss flat and uninformative near the failure region.

Edge cases

EC1. What happens to standard-GAN training the moment the discriminator becomes near-perfect (D(real)≈1, D(fake)≈0)?

The generator's gradient collapses toward zero — the discriminator is so confident there's nothing left to push against. This is the classic vanishing-gradient stall WGAN was built to avoid.

EC2. What is the Wasserstein distance between two identical distributions?

Exactly zero — no mass needs to move. This is the well-behaved fixed point the generator is driving toward.

EC3. What if the clip constant

c

is set far too large in WGAN?

The Lipschitz constraint effectively disappears, letting the critic's outputs grow large; the loss stops estimating

W

and gradients can explode — you're back to unstable training.

EC4. What if

c

is set far too small?

The critic becomes over-constrained: its weights are squashed so hard it can barely represent anything, gradients vanish, and learning grinds to a crawl. Clipping's fragility to this single number is its core weakness.

EC5. Real and fake distributions living on thin, low-dimensional manifolds — why is this the normal case, not a rare one?

Natural images occupy a tiny sliver of all possible pixel arrays, so two such slivers almost surely fail to overlap early in training. That's precisely when JS saturates and Wasserstein's smoothness rescues the gradient.

EC6. In the generator's size formula

out = (in - 1) \cdot stride - 2 \cdot padding + kernel

, what does stride=1, padding=0, kernel=1 produce from a 4×4 input?

(4 - 1) \cdot 1 - 0 + 1 = 4

— the size is unchanged, illustrating why such a layer can never upsample toward 64×64.

EC7. If you fed the discriminator a batch that was all real images with BatchNorm on the input layer, what breaks conceptually?

BatchNorm's statistics would be computed over an all-real batch, so at test/mixed time the normalization shifts inconsistently — the model's judgment depends on batch composition rather than the image itself.

Recall Rapid self-check

Which fixed the loss, which fixed the architecture? ::: WGAN fixed the loss (Wasserstein vs JS); DCGAN fixed the architecture (convolutions, batch norm, Adam settings). What does zero gradient from a constant loss mean for learning? ::: The network gets no direction to improve, so training stalls even though the loss "says" it's wrong. Critic output type in WGAN? ::: An unbounded real score, not a probability — no sigmoid.