DCGAN, WGAN, StyleGAN: Worked examples — AI-ML deep dive

This page is a workbench. We take the ideas from the parent topic and we run the numbers — every sign, every corner case, every degenerate input — so that when you meet one of these in an exam or in code, you have already seen it.

Before we start, four tiny reminders written from zero, so no symbol surprises you:

Definition Four symbols we will lean on

$W (P_{r}, P_{g})$ — the Wasserstein distance (also "Earth-Mover distance"). Picture two piles of sand: $P_{r}$ (the real pile) and $P_{g}$ (the generated pile). $W$ is the least total work (mass $\times$ distance) needed to reshape the fake pile into the real pile. Bigger $W$ = piles are further apart.
$D_{J S} (P_{r} ∥ P_{g})$ — the Jensen–Shannon divergence, the quantity vanilla GANs secretly minimize. It measures how much two distributions overlap: it is $0$ when they are identical and rises to a maximum of $lo g 2 \approx 0.693$ when they share no overlap at all. Its explicit formula (mixture $M = \frac{1}{2} (P_{r} + P_{g})$ ) is $D_{J S} = \frac{1}{2} D_{K L} (P_{r} ∥ M) + \frac{1}{2} D_{K L} (P_{g} ∥ M)$ ; but for this page the only fact we need is the plateau at $lo g 2$ for disjoint piles.
$out = (in - 1) \cdot s - 2 p + k$ — the transposed-convolution size formula. in = input side length in pixels, s = stride, p = padding, k = kernel side. It tells you the output side length after one upsampling layer. We earned this in the parent's generator table; here we stress-test it.
$∣ f (x_{1}) - f (x_{2}) ∣ \leq ∣ x_{1} - x_{2} ∣$ — the 1-Lipschitz condition. In plain words: the critic function $f$ is not allowed to rise or fall faster than a 45° ramp. Its output can never change more than the input changes.

The scenario matrix

Every problem this topic can throw lands in one of these cells. The worked examples below are each tagged with the cell they cover.

#	Case class	What makes it tricky	Example
A	Overlapping distributions	$W$ small and positive	Ex 1
B	Disjoint distributions, far apart	JS saturates but $W$ grows	Ex 2
C	Zero / identical distributions	$W = 0$ (degenerate)	Ex 3
D	Sign of the critic loss	Which way does the number move?	Ex 4
E	Lipschitz violation	Unbounded critic breaks everything	Ex 5
F	Weight-clipping arithmetic	The blunt $[- c, c]$ instrument	Ex 6
G	Transposed-conv size (normal)	Doubling resolution	Ex 7
H	Transposed-conv degenerate (stride 1)	Resolution doesn't grow	Ex 8
I	Real-world word problem	Translate story → $W$	Ex 9
J	Exam twist	Combine JS-vs- $W$ + gradient	Ex 10
K	WGAN-GP gradient penalty	Softer Lipschitz enforcement	Ex 11

Example 1 — Cell A: two overlapping 1-D piles

Step 1 — Understand the pile. Each distribution is a single grain of mass $1$ . $P_{r}$ 's grain sits at $3$ , $P_{g}$ 's grain sits at $1$ . Why this step? Wasserstein is about moving mass; you must first see where the mass actually is.

Step 2 — There is only one way to couple them. With one grain each, the only transport plan moves that whole grain from $1$ to $3$ . Why this step? $W$ is an infimum over all couplings — but with single points there is exactly one coupling, so the infimum is trivial.

Step 3 — Cost = mass × distance. $W = 1 \times ∣3 - 1∣ = 2$ Why this step? This is literally the definition $E_{(x, y) \sim γ} [∣ x - y ∣]$ with a degenerate $γ$ .

Verify: Units are "distance" (position units), correct for a distance metric. Move the grain the other direction and you'd get $∣1 - 3∣ = 2$ — symmetric, as a distance must be. ✓

Figure guide: the horizontal axis is $θ$ (how far apart the piles are); the vertical axis is the divergence value. The amber V-shaped curve is $W = ∣ θ ∣$ — it rises steadily as the piles separate. The flat cyan horizontal line is $D_{J S}$ , pinned at $lo g 2$ for every non-zero $θ$ .

Step 1 — Wasserstein. Same single-grain logic as Ex 1: $W (P_{r}, P_{g}) = ∣ θ ∣$ Why this step? One grain must travel distance $∣ θ ∣$ . This is the amber V-shaped curve — steadily rising, never flat.

Step 2 — Jensen–Shannon. When the two grains sit on different points (any $θ \neq = 0$ ), the distributions share no overlap, so $D_{J S} (P_{r} ∥ P_{g}) = lo g 2 \approx 0.6931 (θ \neq = 0), D_{J S} = 0 (θ = 0)$ Why this step? JS measures overlap. No overlap ⇒ it hits its maximum constant $lo g 2$ and stays there no matter how far apart. That's the flat cyan horizontal line — completely level.

Step 3 — Take the slope (gradient) of each.

$\frac{d W}{d θ} = sign (θ)$ — always $\pm 1$ , never zero (except the kink at 0). Points the generator home.
$\frac{d D _{J S}}{d θ} = 0$ for all $θ \neq = 0$ . Says nothing.

Why this step? Training is following a gradient. A zero gradient = no learning signal.

Verify: At $θ = 2$ : $W = 2$ (matches the moving-a-grain intuition of Ex 1), $D_{J S} = lo g 2$ . Slope of $W$ is $+ 1$ ; slope of $D_{J S}$ is $0$ . ✓ This is the entire reason WGAN exists.

Example 3 — Cell C: the degenerate zero case

Step 1 — Couple every grain to itself. Choose the transport plan $γ$ that maps each point to the same point ( $x \mapsto x$ ). Why this step? We need a valid coupling to upper-bound the infimum; the identity map is always valid when the marginals are equal.

Step 2 — Distance moved is zero. $W = E [∣ x - x ∣] = 0$ Why this step? No grain moves, so total work is $0$ , and $W \geq 0$ always — an infimum of $0$ is the minimum.

Verify: A distance metric must satisfy $W (P, P) = 0$ . ✓ Sanity: the critic loss $E [f (x)] - E [f (y)]$ also $\to 0$ when both expectations sample the same distribution. ✓

Example 4 — Cell D: the sign of the critic loss

Step 1 — Wasserstein estimate is the difference. $W = E_{r} [f] - E_{g} [f] = 2.0 - (- 1.5) = 3.5$ Why this step? Kantorovich–Rubinstein duality says the maximized difference is the Wasserstein distance estimate.

Step 2 — The training loss is the negation. $L_{D} = - W = - 3.5$ Why this step? Optimizers descend, so "maximize $W$ " is coded as "minimize $- W$ ".

Step 3 — Interpret the sign. A large positive $W$ ⇒ a very negative $L_{D}$ ⇒ critic is confidently separating real from fake. Why this step? This is exactly the sign-flip students trip over: a "good" critic shows a negative loss number, which looks alarming but is correct.

Verify: $- (2.0 - (- 1.5)) = - 3.5$ . If critic couldn't tell them apart, both scores equal ⇒ $W = 0 \Rightarrow L_{D} = 0$ , matching Ex 3's $W = 0$ . ✓

Example 5 — Cell E: a Lipschitz violation blows up

Figure guide: horizontal axis is the input $x$ , vertical axis is the critic output $f (x) = k x$ . Three straight lines through the origin: the cyan 45° line is $k = 1$ (the legal ceiling), the dashed amber line is $k = 5$ (too steep — illegal), and the faint white line is $k = 0.4$ (legal but gentle). Any line steeper than the cyan one violates 1-Lipschitz.

Step 1 — Compute the unconstrained gap. $W_{raw} = f (0) - f (2) = 0 - 2 k = - 2 k$ Why this step? With no ceiling on $k$ , multiply the critic's weights by $1000$ and the estimate scales by $1000$ — the number stops meaning "distance." Steeper amber lines in the figure keep pushing the gap larger without bound.

Step 2 — Why the critic wants a negative slope. The critic maximizes $E_{r} [f] - E_{g} [f] = f (0) - f (2) = - 2 k$ . This expression is largest when $k$ is as negative as allowed. So the optimal critic picks $k = - 1$ (steepest legal downhill ramp), giving $- 2 (- 1) = + 2$ . Why this step? The supremum in Kantorovich–Rubinstein is a maximization. Since real mass sits left of fake mass here, the critic scores left-points higher — a downward-sloping $f$ . This is the sign detail students skip: the optimal $f$ is $f (x) = - x$ , not $+ x$ .

Step 3 — At the optimal legal slope $k = - 1$ : $W = f (0) - f (2) = 0 - (- 2) = 2 = ∣ θ ∣$ Why this step? The constrained optimum recovers the true Wasserstein distance from Ex 2. The constraint isn't red tape — it's what makes the number a distance. (The absolute value in Ex 2 is exactly this: $W = ∣ θ ∣$ picks whichever slope sign, $+ 1$ or $- 1$ , makes the difference positive.)

Verify: With $k = - 1$ , $W = 0 - (- 2) = 2 = W$ from Ex 2. With $k = - 5$ (illegal) you'd get a fake " $10$ ." The constraint restores meaning; the negative slope maximizes the objective. ✓

Example 6 — Cell F: weight-clipping arithmetic

Step 1 — Apply $clip (w, - 0.01, 0.01)$ element-wise. $w \to [0.01, - 0.008, 0.01, 0.004]$

$0.03 > 0.01 \Rightarrow 0.01$ (pinned high)
$- 0.008$ inside $[- 0.01, 0.01] \Rightarrow$ unchanged
$0.011 > 0.01 \Rightarrow 0.01$ (pinned high)
$0.004$ inside $\Rightarrow$ unchanged

Why this step? Clipping is a crude enforcement of Lipschitz: bound the weights ⇒ bound the gradient (chain rule) ⇒ roughly bound the slope.

Step 2 — Count the pinned. Two of four weights hit the boundary. Why this step? This exposes clipping's known flaw: weights pile up at $\pm c$ , wasting capacity — the reason WGAN-GP (Ex 11) later replaced clipping with a gradient penalty.

Verify: Only entries with $∣ w ∣ > 0.01$ change: that's $0.03$ and $0.011$ → exactly $2$ pinned. ✓

Example 7 — Cell G: transposed-conv doubles a feature map

Figure guide: on the left, a coarse cyan $8 \times 8$ grid (the input feature map). An amber arrow labelled ConvTranspose k=4 s=2 p=1 points right to a finer white $16 \times 16$ grid (the output). The picture shows the side length doubling as the layer upsamples.

Step 1 — Plug into the size formula. $out = (in - 1) \cdot s - 2 p + k = (8 - 1) \cdot 2 - 2 \cdot 1 + 4$ Why this step? This is the exact rule for a ConvTranspose2d output side; the figure shows the $8 \times 8$ tiles fanning out into a $16 \times 16$ canvas.

Step 2 — Arithmetic. $out = 14 - 2 + 4 = 16$ Why this step? Confirms the "double the resolution" claim: $8 \to 16$ .

Verify: $16 = 2 \times 8$ . The full chain $4 \to 8 \to 16 \to 32 \to 64$ each obeys this same formula. ✓

Example 8 — Cell H: the degenerate stride-1 mistake

Step 1 — Plug in. $out = (4 - 1) \cdot 1 - 2 \cdot 1 + 4 = 3 - 2 + 4 = 5$ Why this step? Same formula, only $s$ changed. Notice the output side went $4 \to 5$ : it crawls up by one pixel, not doubling.

Step 2 — Diagnose. To reach $64 \times 64$ from $4 \times 4$ you'd need dozens of these crawling layers — practically the generator gets stuck near its seed resolution. Why this step? This is exactly the parent's "Mistake 2": stride-1 for upsampling. The fix is $s = 2$ (Ex 7).

Verify: $5 \neq = 8$ , so stride-1 does not double resolution. To double you must use $s = 2$ . ✓

Example 9 — Cell I: a real-world word problem

Step 1 — Match nearest-to-nearest (the optimal coupling). Move $0.5$ t from $1 \to 2$ (distance $1$ ) and $0.5$ t from $5 \to 4$ (distance $1$ ). Why this step? $W$ is the infimum — the cheapest plan. Sorting both piles and matching in order is optimal on the line.

Step 2 — Total cost = Σ mass × distance. $W = 0.5 \cdot 1 + 0.5 \cdot 1 = 1.0 tonne\cdotkm$ Why this step? Direct evaluation of $E_{γ} [∣ x - y ∣]$ with the chosen $γ$ .

Step 3 — Check the crossover plan is worse. Moving $1 \to 4$ (dist $3$ ) and $5 \to 2$ (dist $3$ ) costs $0.5 \cdot 3 + 0.5 \cdot 3 = 3.0$ — three times more. Why this step? Confirms our plan is the infimum, not just a plan.

Verify: Min cost $= 1.0$ ; crossover $= 3.0 > 1.0$ . Units tonne·km, a valid "work" quantity. ✓

Example 10 — Cell J: exam twist (combine everything)

Step 1 — (a) Diagnose the false conclusion. A zero JS gradient at $θ = 3$ is the saturation trap (Ex 2), not convergence. The true remaining distance is $W = ∣ θ ∣ = 3 \neq = 0.$ Why this step? Convergence means $W = 0$ (Ex 3). Here $W = 3$ , so the model is far from done — JS just went blind. Answer: False.

Step 2 — (b) Wasserstein estimate. $W = E_{r} [f] - E_{g} [f] = 0.9 - (- 2.6) = 3.5$ Why this step? Kantorovich–Rubinstein again (Ex 4): the maximized difference of critic scores is the Wasserstein estimate.

Step 3 — (b) Code loss (the sign flip). $L_{D} = - W = - 3.5$ Why this step? Optimizers descend, so "maximize $W$ " is coded as "minimize $- W$ " (Ex 4). A large positive $W$ ⇒ a very negative $L_{D}$ ⇒ the critic is separating real from fake well.

Verify: $W = 3$ from $∣ θ ∣$ ; $W = 0.9 + 2.6 = 3.5$ ; $L_{D} = - 3.5$ . Consistent with Ex 2 and Ex 4. ✓

Example 11 — Cell K: WGAN-GP gradient penalty

Step 1 — Understand why gradient norm $= 1$ is the target. A 1-Lipschitz function has slope magnitude at most $1$ everywhere; the optimal WGAN critic actually has slope exactly $1$ along the transport path. So instead of clipping weights (Ex 6, the blunt instrument), we nudge $∥\nabla f ∥_{2}$ toward $1$ . Why this step? This is the smooth replacement for clipping — it enforces the same 1-Lipschitz idea (Ex 5) without pinning weights to $\pm c$ .

Step 2 — Plug the measured norm into the penalty. $λ (∥\nabla f ∥_{2} - 1)^{2} = 10 (1.4 - 1)^{2} = 10 \times (0.4)^{2} = 10 \times 0.16 = 1.6$ Why this step? Direct evaluation. The gap $(1.4 - 1) = 0.4$ is squared (so over- and under-shoot both cost) and scaled by $λ = 10$ .

Step 3 — Check the degenerate case. If the slope were exactly $1$ , the penalty is $10 (1 - 1)^{2} = 0$ — no cost, matching the ideal 1-Lipschitz critic. Why this step? Confirms the penalty rewards precisely the constraint we want, and never punishes a perfect critic.

Verify: $10 (1.4 - 1)^{2} = 1.6$ ; at norm $= 1$ the penalty is $0$ . Units are "loss" (same as critic loss), so it adds cleanly to $L_{D}$ . ✓

Recall Quick self-test

Wasserstein of $P_{r}$ at $x = 3$ vs $P_{g}$ at $x = 1$ ::: $2$ $D_{J S}$ between two disjoint point-masses ::: $lo g 2 \approx 0.693$ Slope of $W (θ) = ∣ θ ∣$ for $θ > 0$ ::: $+ 1$ (never zero — usable gradient) Transposed-conv output for in=8,k=4,s=2,p=1 ::: $16$ Same layer but stride=1, in=4 ::: $5$ (does NOT double — the bug) Critic scores real $= 2.0$ , fake $= - 1.5$ : the code loss $L_{D}$ ::: $- 3.5$ Clip $[0.03, - 0.008, 0.011, 0.004]$ to $\pm 0.01$ : how many pinned ::: $2$ Optimal critic slope for real-left-of-fake ( $f (x) = k x$ , 1-Lipschitz) ::: $k = - 1$ WGAN-GP penalty $10 (∥\nabla f ∥ - 1)^{2}$ at norm $1.4$ ::: $1.6$ $W (P, P)$ ::: $0$

Worked examples — DCGAN, WGAN, StyleGAN