DCGAN, WGAN, StyleGAN: Visual walkthrough — AI-ML deep dive

This page builds ONE idea from the ground up: why the ordinary GAN loss stops teaching the generator, and why the Earth-Mover (Wasserstein) distance fixes it. We use nothing but two piles of dirt on a line. No probability jargon is assumed — every symbol is earned before it appears.

Step 1 — Draw the two piles

WHAT. We put every "distribution" onto a single horizontal line so we can literally see it. A distribution is just: where is the sand, and how much is at each spot? Total sand always adds to $1$ (it is a probability — a share of the whole).

$P_{r}$ = the real distribution. Read it as "P-real", the sand pile for genuine data.
$P_{g}$ = the generated distribution. Read it as "P-gen", the sand pile the generator makes right now.

WHY. GAN training only ever compares these two piles. If we can measure "how different are these two piles?" as a single number, we can shrink it. Everything else is machinery for computing that number.

PICTURE. Two narrow spikes, real in yellow, fake in blue, separated by a gap $θ$ (the distance between them). We will move the blue spike toward the yellow one.

Step 2 — What a "loss" has to give us

WHAT. A loss is a function that eats the current situation and spits out one number, "how bad is it?". Call it $L (θ)$ . We then compute its slope with respect to $θ$ and step downhill.

WHY slope, and not just the value? The number alone ("badness = 5") does not tell you which direction to move. The slope — how much $L$ changes when $θ$ nudges — is the arrow that points downhill. In symbols the slope is written

$\frac{d L}{d θ}$

Read $\frac{d L}{d θ}$ as: "if $θ$ increases by a tiny bit, by how much does $L$ increase?" It is the steepness of the loss curve. If this slope is zero, the curve is flat there and we get no direction — training freezes. Hold onto that: flat = dead.

PICTURE. A ball on a curve. Where the curve tilts, the ball rolls; where the curve is flat, the ball just sits.

Step 3 — Measure the gap the JS way

WHAT. The classic GAN loss secretly measures the Jensen–Shannon divergence, written $D_{J S} (P_{r} ∥ P_{g})$ . Read it as "the JS distance between real and fake". "Divergence" just means "a number that is $0$ when the piles are identical and grows when they differ".

WHY it behaves badly here. JS is built from overlap: it asks "at each point on the line, do both piles have sand?" For our two spikes, as long as $θ \neq = 0$ the spikes sit at different points — zero overlap. And here is the trap: whether the gap is tiny ( $θ = 0.01$ ) or huge ( $θ = 100$ ), the overlap is the same — none. So JS returns the same value no matter how far apart they are.

For two non-overlapping spikes the exact value is the constant

$D_{J S} (P_{r} ∥ P_{g}) = lo g 2 \approx 0.693 (for every θ \neq = 0)$

and only at $θ = 0$ does it drop to $0$ .

$lo g 2$ — a fixed number, the natural logarithm of $2$ . Its exact size does not matter; what matters is that it does not change with $θ$ .

PICTURE. A flat plateau at height $lo g 2$ for all $θ \neq = 0$ , then a sudden cliff to $0$ exactly at $θ = 0$ . Flat everywhere the generator actually is.

Step 4 — Measure the gap the Earth-Mover way

WHAT. The Wasserstein-1 distance, written $W (P_{r}, P_{g})$ ("W of real and gen"), asks a completely different question: how much sand-moving work does it take to turn the fake pile into the real pile? Work = (amount of sand) $\times$ (distance you drag it).

WHY this is the right question. Because for our two spikes the answer is dead simple: all the fake sand sits at $θ$ , all the real sand sits at $0$ , so you drag every grain a distance $θ$ . With total sand $= 1$ :

$W (P_{r}, P_{g}) = ∣ θ ∣$

$∣ θ ∣$ — the absolute value of the gap (distance is never negative).
Notice it grows smoothly as the gap grows. Twice as far apart $\Rightarrow$ twice the work. Unlike JS, it reports the distance itself.

PICTURE. A little conveyor moving the blue spike onto the yellow spike; the length of the drag is exactly $θ$ .

Step 5 — Put both slopes side by side

WHAT. We now differentiate both loss curves and read off their arrows.

WHY this is the punchline. Training follows the slope (Step 2). So the loss that has a usable slope is the loss that trains.

\qquad\text{vs}\qquad \underbrace{\frac{d}{d\theta}\,|\theta| = \operatorname{sign}(\theta)}_{\text{constant nonzero arrow}}$$ - $\operatorname{sign}(\theta)$ — equals $+1$ if $\theta>0$, $-1$ if $\theta<0$. It always points *back toward $0$*, i.e. toward matching the real pile. - The JS slope is $0$: no arrow. - The Wasserstein slope is $\pm 1$: a clean, constant push toward the goal — *even when the piles do not overlap.* **PICTURE.** The two curves overlaid: JS a flat shelf (dead ball), $|\theta|$ a clean V-valley (ball rolls straight to the bottom). ![[deepdives/dd-ai-ml-4.5.09-d2-s05.png]] > [!example] Numbers at $\theta = 3$ > - JS loss value $=\log 2\approx 0.693$; JS slope $=0$ → generator learns **nothing**. > - Wasserstein loss value $=3$; slope $=+1$ → step *left* by $\alpha\cdot 1$, gap shrinks. > - At $\theta = 0.5$: JS still $\log 2$ (slope $0$); Wasserstein $=0.5$ (slope still $+1$). The push never dies until the gap is closed. --- ## Step 6 — How do we actually *compute* $W$ in a network? **WHAT.** We cannot enumerate all transport plans inside a neural net. A theorem called **Kantorovich–Rubinstein duality** rewrites $W$ as a *maximization* we can train: $$W(P_r,P_g)=\sup_{\Vert f\Vert_L\le 1}\;\mathbb{E}_{x\sim P_r}[f(x)]-\mathbb{E}_{x\sim P_g}[f(x)]$$ - $f$ — a **critic** function (a neural net) that scores each point with a single number. - $\mathbb{E}_{x\sim P_r}[f(x)]$ — the average score the critic gives to **real** sand. - $\mathbb{E}_{x\sim P_g}[f(x)]$ — the average score it gives to **fake** sand. - $\sup$ — "supremum", the largest achievable value: the critic *tries to separate* real from fake as hard as it can. - $\Vert f\Vert_L\le 1$ — the **1-Lipschitz** leash: $f$ may rise by at most $1$ for every $1$ you move along the line, $|f(a)-f(b)|\le|a-b|$. **WHY the leash matters (degenerate case!).** Without it, the critic could scream "real $=+\infty$, fake $=-\infty$" by scaling its weights up, and the difference would blow up to infinity — a meaningless loss with no gradient signal, exactly the failure we escaped. The Lipschitz cap forces the critic to grow **gently**, so its slope reflects the true geometry — the sand-moving distance. **PICTURE.** A critic line climbing at most $45^\circ$ (slope $\le 1$) from the fake spike up to the real spike; its total rise across the gap *equals* the transport cost $|\theta|$. ![[deepdives/dd-ai-ml-4.5.09-d2-s06.png]] > [!mistake] Forgetting the Lipschitz leash > **Why it feels right:** "Let the critic be as powerful as possible." > **Why it's wrong:** an unconstrained critic sends the loss to $\pm\infty$ and its gradient becomes garbage — the very disease WGAN cures returns. > **The crude fix (original WGAN):** clip weights to $[-c,c]$ with $c=0.01$ so the net cannot get steeper than slope $1$. Blunt, but it keeps the leash on. --- ## Step 7 — The degenerate case that ties it together **WHAT.** Check the goal state $\theta = 0$: piles identical. **WHY.** A good loss must be $0$ *and* have the generator stop naturally there. - $W = |0| = 0$ ✓ (no sand to move). - $D_{JS} = 0$ ✓ (they overlap perfectly). Both agree **at the destination**. They disagree only *on the way there* — and the way there is 99% of training. JS is flat on the whole journey and only correct at the finish line you can never reach without a slope; Wasserstein hands you a downhill slope the entire trip. **PICTURE.** Zoom on $\theta=0$: both curves touch $0$, but JS arrives via a vertical cliff (no approach slope) while $|\theta|$ arrives via a smooth V (a slope guiding you in). ![[deepdives/dd-ai-ml-4.5.09-d2-s07.png]] Reveal drills: Why does the standard GAN loss freeze on non-overlapping piles? ::: Because JS saturates at the constant $\log 2$, so its slope $\dfrac{dD_{JS}}{d\theta}=0$ — no direction to descend. What single number does Wasserstein report for our two spikes? ::: The gap itself, $W=|\theta|$. What is the Wasserstein slope, and why is it good? ::: $\operatorname{sign}(\theta)=\pm1$, a constant nonzero push toward $\theta=0$ from any distance. Why must the critic $f$ be 1-Lipschitz? ::: To stop it scaling to $\pm\infty$; the leash makes its slope encode the true transport distance and keeps gradients finite. --- ## The one-picture summary ![[deepdives/dd-ai-ml-4.5.09-d2-s08.png]] One frame, whole story: two spikes on a line (Step 1); JS gives a flat plateau at $\log 2$ with zero slope (dead ball); Wasserstein gives a V-valley equal to $|\theta|$ with a constant $\pm1$ slope (rolling ball); the critic climbs at slope $\le 1$ so its rise measures that gap. > [!recall]- Feynman retelling — say it like you'd tell a friend > You've got two little heaps of sand on a ruler: the *real* heap at zero and the *fake* heap somewhere out at distance $\theta$. Training just means shoving the fake heap onto the real one, and to do that your loss must whisper *which way and how far*. > > The old GAN loss (JS) only checks "do the heaps sit on the same spot?" As long as they don't, it shrugs and says the same bored number, $\log 2$, whether they're a hair apart or a mile apart. A flat number has no tilt, and gradient descent lives on tilt, so the generator just sits there — a compass frozen on "nope". > > The Wasserstein loss instead asks "how much shoveling to move fake onto real?" That answer *is* the distance $\theta$: farther apart, bigger number, and its slope is a steady $\pm1$ pointing straight home. Since we can't shovel sand inside a neural net, a theorem lets a **critic** net do the measuring instead — but only if we keep it on a leash (1-Lipschitz, slope $\le 1$), otherwise it cheats to infinity and we're back to a useless flat loss. Leash on, the critic's gentle climb across the gap equals the sand-moving cost, and the generator finally has a real hill to roll down. --- [[4.5.09 DCGAN, WGAN, StyleGAN (Hinglish)|← Read this in Hinglish]]