Exercises — DCGAN, WGAN, StyleGAN

• (a) LeakyReLU → cures the dying ReLU problem; negative pre-activations still leak a gradient, so neurons keep learning.
• (b) tanh → squashes output to $[-1,1]$, matching the range of images that were normalised to $[-1,1]$.
• (c) no BatchNorm on discriminator input → recall BatchNorm rescales activations across the batch; on the input layer that would mix real and fake samples together, leaking information across the batch, so each image can no longer be judged on its own.
• (d) strided convolutions → let the network learn its own down/upsampling instead of a fixed pooling rule, and enable clean upsampling in the generator.

The critic $f_w$ must be 1-Lipschitz: $|f_w(x_1)-f_w(x_2)|\le |x_1-x_2|$ for all inputs. This bounds how fast the critic's output can change. Where this requirement comes from (the Kantorovich–Rubinstein duality, in plain words): the Wasserstein distance is originally defined as a hard "move all the dirt for minimal total work" problem. A classic theorem in optimal transport says that exact same number can instead be computed by searching over functions $f$ and taking ${\sup }_f({\mathbb{E}}_{P_r}[f]-{\mathbb{E}}_{P_g}[f])$ — but only if we restrict $f$ to functions whose slope never exceeds 1 (1-Lipschitz). That restriction is what makes the two viewpoints equal, which is why the critic must obey it. A standard discriminator has no such requirement (it just outputs a probability through a sigmoid).

• (a) The mapping network — a small multi-layer perceptron that maps $\mathbf{z}$ (a plain Gaussian sample) into the intermediate latent code $\mathbf{w}$. Why bother: $\mathbf{z}$ lives on a fixed round Gaussian, which forces unnatural entanglement of features; $\mathbf{w}$ is free to unwarp into a space where directions line up with real, disentangled attributes.
• (b) Coarse layers (low resolution, e.g. $4\times 4$–$8\times 8$) control big-picture attributes — pose, face shape, hairstyle. Fine layers (high resolution) control small details — skin texture, freckles, micro-colour. This split is exactly the independent control the later Level-5 synthesis problem exploits.

Apply the output-size formula (with $\text{output\_padding}=0$) using $\text{in}=4$, $\text{stride}=2$, $\text{padding}=1$, $\text{kernel}=4$: $\text{out}=(4-1)\times 2-2\times 1+4+0=6-2+4=8$ Output is $8\times 8$. The layer doubles resolution — exactly what we want when climbing $4\to 8\to 16\to 32\to 64$, the ladder drawn in Figure 2.

Output units = $512\times 4\times 4=8192$. Weight count = (input dim) $\times$ (output dim) $=100\times 8192=819200$ weights. This single dense layer is the only fully connected layer allowed — it builds the initial $4\times 4$ "seed" grid, after which everything is convolutional.

The critic descends its loss: $w\leftarrow w-\alpha \cdot {\nabla }_w{L}_D$. (a) $w=0.008-0.00005\times 3.0=0.008-0.00015=0.00785$. (b) Clip to $[-0.01,0.01]$: since $0.00785$ is inside the range, it is unchanged → $0.00785$. Why clip at all (tie back to WGAN): the critic must stay 1-Lipschitz — its slope must never exceed 1. By the chain rule the critic's slope is bounded by a product of its weights, so if we force every weight to live inside $[-c,c]$ we bound the slope and (crudely) enforce the Lipschitz constraint that makes the Wasserstein estimate — and therefore training — stable. Here the clip does not bite because $0.00785$ is already inside the box; it only activates when a weight leaves $[-c,c]$.

(a) For disjoint supports the JS divergence saturates at the constant $\ln 2\approx 0.693$ (natural log), for every nonzero $\theta$. It is flat. Figure 3 shows this as the horizontal dashed line. (b) With all mass at $0$ and all at $\theta$, the only coupling moves that single grain a distance $|\theta |$, so $W(P_r,P_g)=|\theta |$ — the V-shaped magenta line in Figure 3. (c) Differentiate: $\frac{\partial D_{JS}}{\partial \theta }=0(\theta \ne 0),\frac{\partial W}{\partial \theta }=\text{sign}(\theta ).$ JS gives zero gradient — the generator learns nothing about which way to move $\theta$. Wasserstein gives $\pm 1$, a constant nonzero push toward $\theta =0$. This is precisely why WGAN keeps training when distributions barely overlap: its loss slopes toward the answer instead of sitting on a plateau.

In WGAN the critic's job is to estimate the Wasserstein distance: the generator's gradient is only trustworthy once $f_w$ is close to the true supremum ${\sup }_{\Vert f{\Vert }_L\le 1}\mathbb{E}[f(x_r)]-\mathbb{E}[f(x_g)]$. A poorly optimised critic gives a biased distance estimate, so we train it $\sim 5$ steps to near-optimality before each generator step. In a vanilla GAN the opposite holds: a near-perfect discriminator drives $D(G(z))\to 0$, and the generator's gradient (which contains a $1/D$-type factor) collapses — the compass goes dead. The difference is that WGAN's critic never saturates (no sigmoid, no $\ln 2$ plateau), so "stronger critic" means "better distance estimate", not "vanishing signal".

The key is the normalisation step happens first, every layer. Because AdaIN divides out the current mean $\mu (x)$ and standard deviation $\sigma (x)$ before applying the new $y_s,y_b$, it erases the statistics injected by the previous layer and stamps in fresh ones from $\mathbf{w}$.

• Consequence: the style controlling one resolution is localised to that resolution — it does not leak forward as an accumulating bias.
• This is what lets "style mixing" work: feed ${\mathbf{w}}_A$ to coarse layers and ${\mathbf{w}}_B$ to fine layers, and you get person A's pose with person B's skin texture, because each block's contribution is overwritten cleanly at the next normalisation. Contrast with simply adding a style vector: additions accumulate, so early-layer choices would bleed into and entangle later ones — exactly the disentanglement AdaIN's normalise-then-modulate order avoids.

Doubling from $4$: $4\to 8\to 16\to 32\to 64\to 128$. That is 5 doublings, so 5 transposed-conv layers. Halving channels through the hidden layers $512\to 256\to 128\to 64\to 32$, and then the final layer replaces the halved value with 3 (the RGB requirement — it is not $32/2=16$):

Every layer except the last: BatchNorm → ReLU (recall BatchNorm keeps the activations well-scaled so this deep upsampling stack stays trainable). Last layer: tanh. Trace the widths in Figure 2 — the $128$ schedule is the same ladder with one extra rung. (Contrast with the parent's $64\times 64$ generator, which needed only 4 layers.)

They are not the same. Lipschitzness bounds the slope, not the range. Consider $f(x)=\tanh (1000x)$: its output is always inside $[-1,1]$ (bounded!), yet near $x=0$ its slope is $1000\times {\text{sech}}^2(0)=1000$, so its Lipschitz constant is $1000\gg 1$.

• Bounded output $\ne$ bounded derivative.
• Weight clipping instead bounds the weights, which by the chain rule bounds the gradient $|f^{\prime }|$ — i.e. the slope — which is what 1-Lipschitz actually requires. So the student's activation-clipping rule fails: a critic could still change arbitrarily fast between two nearby inputs. (This is exactly why later work replaced clipping with a gradient penalty that pins $\Vert \nabla f\Vert \approx 1$ directly.)

(a) Deriving the critic's orientation — this is the WHY that fixes the sign. The critic maximises $f(0)-f(3)$ over all functions with slope at most 1. On the interval $[0,3]$ the biggest possible gap between $f(0)$ and $f(3)$ under a slope-$1$ cap is achieved when $f$ decreases as fast as allowed, i.e. slope $=-1$, giving $f(x)=-x$ (up to a constant). Check the alternative $f(x)=+x$: it gives $f(0)-f(3)=0-3=-3<0$, so it would minimise, not maximise — the critic would never choose it. Hence the optimiser converges to $f(x)=-x$, and the objective value is $f(0)-f(3)=0-(-3)=+3=|\theta |=W$ ✓. The negative orientation is forced by the maximisation, not chosen by hand. (b) With the correctly-oriented critic $f(x)=-x$: $L_G=-f(\theta )=-(-\theta )=\theta$, so $\partial L_G/\partial \theta =+1$. Gradient descent moves $\theta \leftarrow \theta -\alpha (+1)=\theta -\alpha$: fake mass slides down toward 0. (c) Each step subtracts $\alpha$ from $\theta$, monotonically decreasing $|\theta |$ until $\theta =0$ where real and fake coincide — the training goal. The key mastery point: because $f$ has slope exactly $\pm 1$ everywhere, the gradient magnitude stays $1$ the whole way, never vanishing regardless of how far apart the piles started.

• From DCGAN: the convolutional generator/discriminator backbone with strided (transposed) convolutions and BatchNorm — it gives spatially-coherent upsampling from a seed to high resolution. Without it you have no scalable image network at all.
• From WGAN: the Wasserstein/critic loss — the disjoint early distributions would kill a JS-based generator's gradient ($\ln 2$ plateau), whereas Wasserstein keeps a meaningful slope, so training starts at all.
• From StyleGAN: the style-based generator injecting the intermediate code $\mathbf{w}$ at multiple resolutions via AdaIN — coarse layers control pose, fine layers control skin texture, giving the requested independent control the other two cannot provide. Necessity: DCGAN gives the how to build, WGAN gives the how to train stably, StyleGAN gives the how to control. Drop any one and the requirement (high-res and stable and controllable) fails.

Prediction: the face keeps person A's pose and overall shape (round face, front-facing) but wears person B's fine skin detail and freckles. Justification: coarse layers ($\le 16^2$) receive ${\mathbf{w}}_A$, so the big-picture geometry is stamped in there. When we reach $32^2$, AdaIN first normalises away the running statistics and then re-modulates with ${\mathbf{w}}_B$'s scale/shift — this overwrites only the fine-scale style, and because normalisation is re-applied at every subsequent layer, ${\mathbf{w}}_B$ owns all the high-frequency detail without disturbing the coarse geometry already realised. This clean separation is only possible because AdaIN normalises before modulating (Exercise 3.3); with additive styles the two codes would entangle and you'd get a muddled blend rather than a crisp A-pose / B-skin combination.

The transposed-conv output size is ==$(\text{in}-1)\times \text{stride}-2\times \text{padding}+\text{kernel}+\text{output\_padding}$. JS divergence between disjoint distributions saturates at $\ln 2$== (natural log $\approx 0.693$), giving zero gradient. Wasserstein between point masses at $0$ and $\theta$ equals ==$|\theta |$. The WGAN critic must be 1-Lipschitz== (bounded slope, not bounded output). StyleGAN maps $\mathbf{z}$ to $\mathbf{w}$ via the mapping network, and injects $\mathbf{w}$ per-layer via AdaIN. AdaIN normalises before modulating, which is why each layer's style overwrites rather than accumulates. "1-Lipschitz" means the graph's slope never exceeds ::: $1$