DCGAN, WGAN, StyleGAN

Overview

These three architectures represent evolutionary milestones in GAN training stability and image quality. Each solved critical problems that plagued vanilla GANs: DCGAN established architectural guidelines for stable training, WGAN fixed the broken loss function mathematically, and StyleGAN revolutionized fine-grained control over generated features.

DCGAN (Deep Convolutional GAN)

WHY each guideline matters:

Strided convolutions: Letting the network learn its own spatial downsampling/upsampling is more flexible than fixed pooling. In generator, transposed convolutions upsample feature maps (e.g., 4×4 → 8×8 → 16×16 → ... → 64×64).
Batch normalization: Normalizes layer inputs to zero mean, unit variance. This prevents gradient explosion/vanishing by keeping activations in reasonable ranges. Why not on input/output? Input is already normalized images, output needs full color range.
No FC layers: Fully connected layers lose spatial structure. Keeping everything convolutional preserves locality—nearby pixels influence each other, not random global mixing.
Activation choices: ReLU in generator encourages sparse, efficient representations. LeakyReLU in discriminator prevents "dying ReLU" problem (gradients flow even for negative inputs). Tanh output maps to [-1, 1] matching normalized image range.
Adam hyperparameters: Standard Adam (β₁=0.9) caused training oscillations. Lower momentum (β₁=0.5) reduces oscillation in adversarial setting.

Formula Generator Architecture (for 64×64 images)

Starting from latent vector $\mathbf{z} \in \mathbb{R}^{100}$ :

\begin{align} \mathbf{z} &\xrightarrow{\text{project + reshape}} 512 \times 4 \times 4 \\ &\xrightarrow{\text{ConvTranspose(256, 4×4, stride=2)}} 256 \times 8 \times 8 \\ &\xrightarrow{\text{ConvTranspose(128, 4×4, stride=2)}} 128 \times 16 \times 16 \\ &\xrightarrow{\text{ConvTranspose(64, 4×4, stride=2)}} 64 \times 32 \times 32 \\ &\xrightarrow{\text{ConvTranspose(3, 4×4, stride=2)}} 3 \times 64 \times 64 \end{align}

Each layer (except last) includes: BatchNorm → ReLU. Final layer uses tanh.

WHY this works: Each transpose convolution doubles spatial dimensions while halving channels, progressively refining coarse structure into fine details. The 4×4 kernels with stride 2 ensure smooth upsampling without checkerboard artifacts.

Worked example DCGAN Generator Forward Pass

Input: Random noise $\mathbf{z} = [0.2, -0.5, 0.8, \ldots] \in \mathbb{R}^{100}$ Step 1: Project to 512·4·4 = 8192 dimensions via fully connected layer

output = FC(z) → [8192] 
reshape → [512, 4, 4]

Why this step? Provides initial spatial structure. The 4×4 grid is like a low-resolution "seed" image.

Step 2: First upsampling

ConvTranspose2d(in=512, out=256, kernel=4, stride=2, padding=1)
BatchNorm2d(256)
ReLU()
→ [256, 8, 8]

Why this step? Output size: $\text{out} = (\text{in}-1) \times \text{stride} - 2 \times \text{padding} + \text{kernel} = (4-1) \times 2 - 2 \times 1 + 4 = 8$ . Doubles resolution, reduces channels (network starts broad and narrows).

Step 3-5: Repeat upsampling

[256, 8, 8] → [128, 16, 16] → [64, 32, 32] → [3, 64, 64]

Final output: 3-channel RGB image, tanh activation maps to [-1, 1] range.

WGAN (Wasserstein GAN)

Intuition The Loss Function Was Broken All Along

Standard GAN loss is equivalent to minimizing the Jensen-Shannon divergence $D_{JS}(P_r \| P_g)$ . Problem: when real and fake distributions don't overlap (common early in training, since images live on low-dimensional manifolds), $D_{JS}$ is not infinite—it saturates at the constant $\log 2$ . A constant has zero gradient, so the generator gets no learning signal. It's like a compass that only says "you're not there yet" but never points a direction.

Wasserstein distance (Earth Mover's Distance) measures "how much work to move one distribution to match another." Crucially, it stays finite and varies smoothly even on disjoint supports (it just equals the average distance you must move mass), so it provides meaningful gradients everywhere. Think of pushing a pile of dirt: JS divergence says "piles don't overlap → same constant no matter how far apart", Wasserstein says "here's exactly how far to push each grain".

Deriving the trainable form:

By Kantorovich-Rubinstein duality:

W(P_r, P_g) = \sup_{\|f\|_L \leq 1} \mathbb{E}_{x \sim P_r}[f(x)] - \mathbb{E}_{x \sim P_g}[f(x)]

Where $\|f\|_L \leq 1$ means $f$ is 1-Lipschitz continuous: $|f(x_1) - f(x_2)| \leq |x_1 - x_2|$ for all $x_1, x_2$ .

WHY this helps: We can approximate this by training a neural network $f_w$ (the "critic", replacing discriminator) to maximize:

L_{\text{critic}} = \mathbb{E}_{x \sim P_r}[f_w(x)] - \mathbb{E}_{z \sim p(z)}[f_w(G_\theta(z))]

Subject to $f_w$ being 1-Lipschitz. Generator minimizes the negative:

L_{\text{gen}} = -\mathbb{E}_{z \sim p(z)}[f_w(G_\theta(z))]

WHY Lipschitz matters: Without the constraint, $f_w$ could grow unbounded (just multiply weights by 1000), making loss meaningless. 1-Lipschitz keeps the critic "bounded" and forces it to learn real structure.

Formula WGAN Training Algorithm

Critic (discriminator) update:

For $n_{\text{critic}}$ iterations (typically 5):

Sample batch $\{x^{(i)}\}$ from real data, $\{z^{(i)}\}$ from noise
Compute loss:

L_D = -\frac{1}{m}\sum_{i=1}^m [f_w(x^{(i)}) - f_w(G_\theta(z^{(i)}))]

Update: $w \leftarrow w - \alpha \cdot \nabla_w L_D$
Clip weights: $w \leftarrow \text{clip}(w, -c, c)$ where $c = 0.01$ Generator update:
Sample batch $\{z^{(i)}\}$ from noise
Compute loss:

L_G = -\frac{1}{m}\sum_{i=1}^m f_w(G_\theta(z^{(i)}))

Update: $\theta \leftarrow \theta - \alpha \cdot \nabla_\theta L_G$ WHY weight clipping? Crude way to enforce 1-Lipschitz constraint. If weights are bounded, gradients are bounded (by chain rule). Limitation: Clipping is a blunt instrument; it biases weights toward clip boundaries and limits capacity.

Worked example WGAN vs. Standard GAN Learning Signal

Scenario: Early training, generated images are blurry noise, real images are sharp faces, and the two distributions barely overlap. Standard GAN (JS divergence) — why gradients vanish:

Discriminator becomes near-perfect: D(real) ≈ 1, D(fake) ≈ 0
Generator's NON-SATURATING objective: L_G = -log D(G(z))
When D(G(z)) ≈ 0  ⇒  the *value* -log D(G(z)) is large,
BUT the gradient ∂L_G/∂θ ∝ (1/D)·D'(...) collapses toward 0
because D is saturated (flat) in this region.

Why this step? The problem is NOT that the loss value is zero—it is that a saturated discriminator has near-flat output, so its gradient with respect to generator parameters vanishes. The generator learns nothing even though it is doing terribly. (This is distinct from the discriminator's own minimax loss value.)

WGAN (Wasserstein distance) — why gradients survive:

f_w(real) = 2.5 (average critic score for real)
f_w(fake) = -1.8 (average critic score for fake)
W ≈ 2.5 - (-1.8) = 4.3
Generator gradient = -∇_θ f_w(G(z)) ← non-zero because f_w
is 1-Lipschitz (never saturates), so its slope stays ≈ O(1).

Why this step? A 1-Lipschitz critic cannot flatten out, so it always supplies a directional signal proportional to how far apart the distributions are. Generator knows "move this direction, you're 4.3 units away."

Common mistake WGAN Training Mistakes

Mistake 1: Using BatchNorm in the WGAN or WGAN-GP critic Why it feels right: "DCGAN used it successfully!" Why it's wrong: BatchNorm couples examples in a batch, so a critic score for one sample depends on the others. In WGAN-GP this is even worse: the gradient penalty is computed per-sample ( $\|\nabla_{\hat{x}} f_w(\hat{x})\|$ ), and BatchNorm makes the output a function of the whole batch, breaking the per-sample gradient penalty. WGAN-GP therefore explicitly forbids BatchNorm in the critic. The fix: Use LayerNorm or InstanceNorm (normalize per-sample), or no normalization at all in the critic.

Mistake 2: Training generator and critic equally (1:1 ratio) Why it feels right: "Fair training for both!" Why it's wrong: Critic must stay ahead to provide an accurate Wasserstein approximation. If generator catches up, critic's scores become meaningless. The fix: Use $n_{\text{critic}} = 5$ (train critic 5× per generator update).

WGAN-GP (Gradient Penalty)

Formula Gradient Penalty Instead of Weight Clipping

Weight clipping is crude. WGAN-GP enforces the Lipschitz constraint via a gradient penalty:

L_{\text{critic}} = \mathbb{E}_{\tilde{x} \sim P_g}[f_w(\tilde{x})] - \mathbb{E}_{x \sim P_r}[f_w(x)] + \lambda\, \mathbb{E}_{\hat{x} \sim P_{\hat{x}}}\big[(\|\nabla_{\hat{x}} f_w(\hat{x})\|_2 - 1)^2\big]

Where $\hat{x} = \epsilon x + (1-\epsilon)\tilde{x}$ with $\epsilon \sim \text{Uniform}[0,1]$ (random interpolation between real and fake).

WHY this works: For 1-Lipschitz functions, the gradient norm ≤ 1 everywhere. Penalizing $(\|\nabla\| - 1)^2$ pushes gradients toward norm exactly 1 (tight Lipschitz bound). WHY interpolations? By the mean-value / Rolle argument, if $f$ violates Lipschitz between $x$ and $\tilde{x}$ , some point on the line between them has $\|\nabla f\| > 1$ . Sampling interpolations catches violations.

Typical: $\lambda = 10$ . Note: because the penalty is per-sample, the critic must not use BatchNorm.

StyleGAN

Definition StyleGAN Architecture Components

Mapping Network: $\mathbf{z} \in \mathcal{Z} \xrightarrow{f} \mathbf{w} \in \mathcal{W}$

8-layer MLP transforms standard Gaussian $\mathbf{z}$ into an intermediate latent space $\mathbf{w}$
WHY? $\mathcal{Z}$ is entangled (changing $z$ affects multiple features at once, and is forced to follow the training data's warped density). $\mathcal{W}$ is learned to be disentangled (each direction controls features more independently).

Learned Affine Transforms ( $A$ ): Each $\mathbf{w}$ is passed through a per-layer learned affine map $A$ to produce a style $\mathbf{y} = (\mathbf{y}_s, \mathbf{y}_b)$ (scale and bias) used by AdaIN.
Adaptive Instance Normalization (AdaIN):

\text{AdaIN}(\mathbf{x}_i, \mathbf{y}) = \mathbf{y}_{s,i}\,\frac{\mathbf{x}_i - \mu(\mathbf{x}_i)}{\sigma(\mathbf{x}_i)} + \mathbf{y}_{b,i}

Where $\mathbf{x}_i$ is the $i$ -th feature-map channel, $\mu(\mathbf{x}_i)$ and $\sigma(\mathbf{x}_i)$ are its spatial mean and std (instance normalization, i.e. per-sample, per-channel), and $(\mathbf{y}_{s,i}, \mathbf{y}_{b,i})$ come from the style.

WHY? Instance-normalizing first erases the previous layer's own statistics, then the style re-imposes new statistics. This is exactly why style at layer $r$ controls only layer $r$ 's features — the effect does not accumulate uncontrollably across layers.

Synthesis Network ( $g$ ):

Starts from a learned constant $4\times4\times512$ tensor (not from $\mathbf{z}$ !).
Progressive upsampling blocks: 4×4 → 8×8 → … → 1024×1024.
Each block: Upsample → Conv → (noise) → AdaIN → Conv → (noise) → AdaIN.
Style $\mathbf{w}$ is injected via AdaIN at every layer (two injections per resolution).

Noise Inputs ( $B$ ): Per-pixel Gaussian noise images are added after each convolution, each scaled by a learned per-channel factor $\mathbf{B}$ .

WHY? Generates stochastic variation (freckles, exact hair placement) without changing global structure, freeing the style/latent codes to encode only high-level attributes.

Style Mixing Regularization: During training, two latent codes $\mathbf{w}_1, \mathbf{w}_2$ are used for a single image, switching from one to the other at a random layer.

WHY? This prevents the network from assuming adjacent styles are correlated, further disentangling the levels and enabling clean style mixing at test time.

Formula StyleGAN Synthesis Block (One Resolution Level

$r$ ) Input: Feature map $\mathbf{x}_{r-1}$ from previous level. Step 1 — Upsample:

\mathbf{x}_{r}^{(0)} = \text{Upsample}(\mathbf{x}_{r-1}) \quad (\text{e.g., } 8\times8 \to 16\times16)

Step 2 — Conv + noise + AdaIN:

\begin{align} \mathbf{x}_{r}^{(1)} &= \text{Conv}_{3\times3}(\mathbf{x}_{r}^{(0)}) \\ \mathbf{x}_{r}^{(2)} &= \mathbf{x}_{r}^{(1)} + \mathbf{B}_r \odot \mathbf{n}_r,\quad \mathbf{n}_r \sim \mathcal{N}(0,1) \\ \mathbf{x}_{r}^{(3)} &= \text{AdaIN}\big(\mathbf{x}_{r}^{(2)},\; A_r(\mathbf{w})\big) \end{align}

Step 3 — Conv + noise + AdaIN:

\begin{align} \mathbf{x}_{r}^{(4)} &= \text{Conv}_{3\times3}\big(\text{LeakyReLU}(\mathbf{x}_{r}^{(3)})\big) \\ \mathbf{x}_{r}^{(5)} &= \mathbf{x}_{r}^{(4)} + \mathbf{B}'_r \odot \mathbf{n}'_r \\ \mathbf{x}_{r}^{(6)} &= \text{AdaIN}\big(\mathbf{x}_{r}^{(5)},\; A'_r(\mathbf{w})\big) \end{align}

Output: $\mathbf{x}_r^{(6)}$ passed to next level.

WHY this structure? Convolutions extract features, noise adds scale-appropriate stochastic detail, AdaIN re-styles the (instance-normalized) features from $\mathbf{w}$ . Two injections per level increases capacity and control.

Worked example Style Mixing in StyleGAN

Goal: Generate an image with person A's pose but person B's hair/face color.

Step 1: Generate two latent codes and map them

z_A = random_normal([512]);  w_A = mapping_network(z_A)  # Source A
z_B = random_normal([512]);  w_B = mapping_network(z_B)  # Source B

Why this step? We must work in the disentangled $\mathcal{W}$ space, not $\mathcal{Z}$ .

Step 2: Inject $w_A$ at coarse layers (4×4, 8×8), $w_B$ at fine layers (≥16×16)

x = learned_constant_4x4
# Coarse structure from A
x = synth_block_4x4(x, style=w_A)   # Pose, rough face shape
x = synth_block_8x8(x, style=w_A)
# Fine appearance from B
x = synth_block_

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, GAN training shuru mein ek nightmare tha—mode collapse, gradients vanish ho jaate the, aur hyperparameters ko tune karna almost impossible lagta tha. Yeh teen architectures—DCGAN, WGAN, aur StyleGAN—basically evolution ke milestones hain jinhone in problems ko step-by-step solve kiya. DCGAN ne prove kiya ki agar aap sahi building blocks use karo—matlab convolutions, batch normalization, aur sahi activation functions—toh training stable ho jaati hai, bhale hi underlying loss function thoda shaky ho. Ise aise samjho jaise ghar banate waqt sahi material use karna, taaki structure collapse na kare.

Ab DCGAN ki core intuition yeh hai ki har architectural choice ka ek solid reason hai. Strided convolutions network ko khud spatial upsampling seekhne dete hain (fixed pooling se zyaada flexible), batch norm activations ko reasonable range mein rakhta hai taaki gradient explode ya vanish na ho, aur fully connected layers hataane se spatial structure preserve rehti hai—nearby pixels ek doosre ko influence karte hain, random global mixing nahi hoti. Generator basically ek chhote 4×4 "seed" se shuru hota hai aur transpose convolutions ke through resolution ko double karta jaata hai (4×4 → 8×8 → ... → 64×64), saath hi channels halve karte jaata hai. Isse coarse structure dheere-dheere fine details mein refine hota hai.

Yeh cheez matter isliye karti hai kyunki DCGAN ne foundation set kiya jispe pura modern generative AI khada hai—aaj jo realistic faces, art, aur images generate hoti hain, unki root yahin hai. Agar aap intuition samajh loge ki kyun batch norm input/output pe nahi lagta, ya kyun β₁=0.5 use karte hain oscillation kam karne ke liye, toh aap sirf formula ratne se aage badhoge—aap actually GAN ka behavior predict aur debug kar paoge. Yeh understanding aage WGAN aur StyleGAN jaise advanced models samajhne ke liye strong base ban jaati hai.

Overview

DCGAN (Deep Convolutional GAN)

WGAN (Wasserstein GAN)

WGAN-GP (Gradient Penalty)

StyleGAN

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Generative Models