DCGAN, WGAN, StyleGAN — Hinglish notes

4.5.9 · AI-ML › Generative Models

Overview

Ye teen architectures GAN training stability aur image quality mein evolutionary milestones represent karti hain. Har ek ne vanilla GANs ki critical problems solve kiye: DCGAN ne stable training ke liye architectural guidelines establish kiye, WGAN ne broken loss function ko mathematically fix kiya, aur StyleGAN ne generated features par fine-grained control ko revolutionize kiya.

DCGAN (Deep Convolutional GAN)

WHY har guideline matter karti hai:

Strided convolutions: Network ko apna spatial downsampling/upsampling khud seekhne dena fixed pooling se zyada flexible hai. Generator mein, transposed convolutions feature maps ko upsample karti hain (e.g., 4×4 → 8×8 → 16×16 → ... → 64×64).
Batch normalization: Layer inputs ko zero mean, unit variance par normalize karta hai. Ye activations ko reasonable ranges mein rakh kar gradient explosion/vanishing prevent karta hai. Input/output par kyun nahin? Input already normalized images hain, output ko full color range chahiye.
No FC layers: Fully connected layers spatial structure kho deti hain. Sab kuch convolutional rakhne se locality preserve hoti hai—nearby pixels ek dusre ko influence karte hain, random global mixing nahin.
Activation choices: Generator mein ReLU sparse, efficient representations encourage karta hai. Discriminator mein LeakyReLU "dying ReLU" problem prevent karta hai (negative inputs ke liye bhi gradients flow karte hain). Tanh output [-1, 1] par map karta hai jo normalized image range se match karta hai.
Adam hyperparameters: Standard Adam (β₁=0.9) training oscillations cause karta tha. Kam momentum (β₁=0.5) adversarial setting mein oscillation reduce karta hai.

Worked example DCGAN Generator Forward Pass

Input: Random noise $z = [0.2, - 0.5, 0.8, \dots] \in R^{100}$ Step 1: Fully connected layer ke zariye 512·4·4 = 8192 dimensions par project karo

output = FC(z) → [8192] 
reshape → [512, 4, 4]

Ye step kyun? Initial spatial structure provide karta hai. 4×4 grid ek low-resolution "seed" image ki tarah hai.

Step 2: Pehla upsampling

ConvTranspose2d(in=512, out=256, kernel=4, stride=2, padding=1)
BatchNorm2d(256)
ReLU()
→ [256, 8, 8]

Ye step kyun? Output size: $out = (in - 1) \times stride - 2 \times padding + kernel = (4 - 1) \times 2 - 2 \times 1 + 4 = 8$ . Resolution double hoti hai, channels reduce hote hain (network broad se shuru ho kar narrow hoti hai).

Step 3-5: Upsampling repeat karo

[256, 8, 8] → [128, 16, 16] → [64, 32, 32] → [3, 64, 64]

Final output: 3-channel RGB image, tanh activation [-1, 1] range par map karti hai.

WGAN (Wasserstein GAN)

Intuition Loss Function Shuru Se Hi Broken Tha

Standard GAN loss Jensen-Shannon divergence $D_{J S} (P_{r} ∥ P_{g})$ minimize karne ke equivalent hai. Problem: jab real aur fake distributions overlap nahin karti (training ke shuruaat mein common, kyunki images low-dimensional manifolds par hoti hain), $D_{J S}$ infinite nahin hota—ye constant $lo g 2$ par saturate ho jaata hai. Ek constant ka zero gradient hota hai, isliye generator ko koi learning signal nahin milta. Ye ek aisi compass ki tarah hai jo sirf kehti hai "tum abhi wahan nahin ho" lekin kabhi direction nahin point karti.

Wasserstein distance (Earth Mover's Distance) measure karta hai "ek distribution ko doosre se match karne ke liye kitna kaam karna hoga." Khas baat ye hai ki ye finite rehta hai aur smoothly vary karta hai disjoint supports par bhi (ye sirf average distance equal hoti hai jo mass ko move karna padta hai), isliye ye har jagah meaningful gradients provide karta hai. Ek dirt pile push karne ki tarah sochlo: JS divergence kehta hai "piles overlap nahin karti → same constant chahe kitni bhi door hon", Wasserstein kehta hai "bilkul exactly kitna door har grain ko push karna hai".

Trainable form derive karna:

Kantorovich-Rubinstein duality se:

W (P_{r}, P_{g}) = ∥ f ∥_{L} \leq 1 sup E_{x \sim P_{r}} [f (x)] - E_{x \sim P_{g}} [f (x)]

Jahan $∥ f ∥_{L} \leq 1$ ka matlab hai $f$ 1-Lipschitz continuous hai: $∣ f (x_{1}) - f (x_{2}) ∣ \leq ∣ x_{1} - x_{2} ∣$ sabhi $x_{1}, x_{2}$ ke liye.

WHY ye help karta hai: Hum ise ek neural network $f_{w}$ ("critic", discriminator ko replace karta hai) train karke approximate kar sakte hain jo maximize kare:

L_{critic} = E_{x \sim P_{r}} [f_{w} (x)] - E_{z \sim p (z)} [f_{w} (G_{θ} (z))]

Subject to $f_{w}$ ke 1-Lipschitz hone ke. Generator negative minimize karta hai:

L_{gen} = - E_{z \sim p (z)} [f_{w} (G_{θ} (z))]

WHY Lipschitz matter karta hai: Constraint ke bina, $f_{w}$ unbounded grow kar sakta hai (bas weights ko 1000 se multiply karo), loss meaningless ho jaata hai. 1-Lipschitz critic ko "bounded" rakhta hai aur use force karta hai real structure seekhne ke liye.

Worked example WGAN vs. Standard GAN Learning Signal

Scenario: Shuruaati training, generated images blurry noise hain, real images sharp faces hain, aur dono distributions barely overlap karti hain. Standard GAN (JS divergence) — kyun gradients vanish hote hain:

Discriminator near-perfect ban jaata hai: D(real) ≈ 1, D(fake) ≈ 0
Generator's NON-SATURATING objective: L_G = -log D(G(z))
When D(G(z)) ≈ 0  ⇒  the *value* -log D(G(z)) is large,
BUT the gradient ∂L_G/∂θ ∝ (1/D)·D'(...) collapses toward 0
because D is saturated (flat) in this region.

Ye step kyun? Problem ye nahin hai ki loss value zero hai—problem ye hai ki ek saturated discriminator ka output near-flat hota hai, isliye generator parameters ke respect mein uska gradient vanish ho jaata hai. Generator kuch nahin seekhta chahe wo bahut bura perform kar raha ho. (Ye discriminator ke apne minimax loss value se alag hai.)

WGAN (Wasserstein distance) — kyun gradients survive karte hain:

f_w(real) = 2.5 (average critic score for real)
f_w(fake) = -1.8 (average critic score for fake)
W ≈ 2.5 - (-1.8) = 4.3
Generator gradient = -∇_θ f_w(G(z)) ← non-zero because f_w
is 1-Lipschitz (never saturates), so its slope stays ≈ O(1).

Ye step kyun? Ek 1-Lipschitz critic flatten out nahin ho sakta, isliye ye hamesha ek directional signal supply karta hai jo proportional hai ki distributions kitni door hain. Generator ko pata chalta hai "is direction mein move karo, tum 4.3 units door ho."

Common mistake WGAN Training Mistakes

Mistake 1: WGAN ya WGAN-GP critic mein BatchNorm use karna Kyun sahi lagta hai: "DCGAN ne ise successfully use kiya tha!" Kyun galat hai: BatchNorm batch mein examples ko couple karta hai, isliye ek sample ka critic score doosron par depend karta hai. WGAN-GP mein ye aur bhi bura hai: gradient penalty per-sample compute hota hai ( $∥ \nabla_{\overset{x}{^}} f_{w} (\overset{x}{^}) ∥$ ), aur BatchNorm output ko pure batch ka function bana deta hai, per-sample gradient penalty ko tod deta hai. WGAN-GP isliye critic mein BatchNorm explicitly forbid karta hai. Fix: LayerNorm ya InstanceNorm use karo (per-sample normalize), ya critic mein bilkul normalization mat karo.

Mistake 2: Generator aur critic ko equally (1:1 ratio) train karna Kyun sahi lagta hai: "Dono ke liye fair training!" Kyun galat hai: Critic ko accurate Wasserstein approximation provide karne ke liye aage rehna chahiye. Agar generator catch up kar le, to critic ke scores meaningless ho jaate hain. Fix: $n_{critic} = 5$ use karo (har generator update par critic 5× train karo).

WGAN-GP (Gradient Penalty)

Formula Weight Clipping Ki Jagah Gradient Penalty

Weight clipping crude hai. WGAN-GP Lipschitz constraint ko ek gradient penalty ke zariye enforce karta hai:

L_{critic} = E_{\tilde{x} \sim P_{g}} [f_{w} (\tilde{x})] - E_{x \sim P_{r}} [f_{w} (x)] + λ E_{\overset{x}{^} \sim P_{\overset{x}{^}}} [(∥ \nabla_{\overset{x}{^}} f_{w} (\overset{x}{^}) ∥_{2} - 1)^{2}]

Jahan $\overset{x}{^} = ϵ x + (1 - ϵ) \tilde{x}$ with $ϵ \sim Uniform [0, 1]$ (real aur fake ke beech random interpolation).

WHY ye kaam karta hai: 1-Lipschitz functions ke liye, gradient norm ≤ 1 har jagah hota hai. $(∥\nabla∥ - 1)^{2}$ ko penalize karna gradients ko exactly norm 1 ki taraf push karta hai (tight Lipschitz bound). WHY interpolations? Mean-value / Rolle argument se, agar $f$ $x$ aur $\tilde{x}$ ke beech Lipschitz violate karta hai, to un dono ke beech line par kisi point par $∥\nabla f ∥ > 1$ hoga. Interpolations sample karna violations pakadta hai.

Typical: $λ = 10$ . Note: kyunki penalty per-sample hai, critic BatchNorm use nahin karna chahiye.

StyleGAN

Definition StyleGAN Architecture Components

Mapping Network: $z \in Z f w \in W$

8-layer MLP standard Gaussian $z$ ko ek intermediate latent space $w$ mein transform karta hai
WHY? $Z$ entangled hai ( $z$ change karna ek saath multiple features affect karta hai, aur training data ki warped density follow karne par majboor hai). $W$ disentangled hone ke liye seekha jaata hai (har direction features ko zyada independently control karta hai).

Learned Affine Transforms ( $A$ ): Har $w$ ko ek per-layer learned affine map $A$ se guzara jaata hai jo ek style $y = (y_{s}, y_{b})$ produce karta hai (scale aur bias) jo AdaIN use karta hai.
Adaptive Instance Normalization (AdaIN):

AdaIN (x_{i}, y) = y_{s, i} \frac{x _{i} - μ ( x _{i} )}{σ ( x _{i} )} + y_{b, i}

Jahan $x_{i}$ $i$ -th feature-map channel hai, $μ (x_{i})$ aur $σ (x_{i})$ uske spatial mean aur std hain (instance normalization, yani per-sample, per-channel), aur $(y_{s, i}, y_{b, i})$ style se aate hain.

WHY? Pehle instance-normalize karna pichhli layer ki apni statistics erase kar deta hai, phir style nayi statistics re-impose karta hai. Yahi reason hai ki layer $r$ par style sirf layer $r$ ke features control karta hai — effect layers across uncontrollably accumulate nahin hota.

Synthesis Network ( $g$ ):

Ek learned constant $4 \times 4 \times 512$ tensor se shuru hota hai ( $z$ se nahin!).
Progressive upsampling blocks: 4×4 → 8×8 → … → 1024×1024.
Har block: Upsample → Conv → (noise) → AdaIN → Conv → (noise) → AdaIN.
Style $w$ har layer par AdaIN ke zariye inject hota hai (har resolution par do injections).

Noise Inputs ( $B$ ): Per-pixel Gaussian noise images har convolution ke baad add hoti hain, har ek ek learned per-channel factor $B$ se scale hoti hai.

WHY? Stochastic variation generate karta hai (freckles, exact hair placement) bina global structure change kiye, style/latent codes ko sirf high-level attributes encode karne ke liye free karta hai.

Style Mixing Regularization: Training ke dauran, ek hi image ke liye do latent codes $w_{1}, w_{2}$ use hote hain, ek random layer par ek se doosre par switch karke.

WHY? Ye network ko ye assume karne se rokta hai ki adjacent styles correlated hain, aur levels ko further disentangle karta hai aur test time par clean style mixing enable karta hai.

Worked example Style Mixing in StyleGAN

Goal: Person A ka pose lekin person B ke hair/face color wali image generate karo.

Step 1: Do latent codes generate karo aur unhe map karo

z_A = random_normal([512]);  w_A = mapping_network(z_A)  # Source A
z_B = random_normal([512]);  w_B = mapping_network(z_B)  # Source B

Ye step kyun? Hume disentangled $W$ space mein kaam karna hai, $Z$ mein nahin.

Step 2: Coarse layers (4×4, 8×8) par $w_{A}$ inject karo, fine layers (≥16×16) par $w_{B}$

x = learned_constant_4x4
# Coarse structure from A
x = synth_block_4x4(x, style=w_A)   # Pose, rough face shape
x = synth_block_8x8(x, style=w_A)
# Fine appearance from B
x = synth_block_