GAN training instability and mode collapse

4.5.8 · AI-ML › Generative Models

Overview

GAN training notoriously difficult hai kyunki hum do networks ko minimax game mein balance karne ki koshish kar rahe hain. Generator discriminator ko fool karne ki koshish karta hai, jabki discriminator generator ke fakes ko pakadne ki koshish karta hai. Jab yeh balance toot ta hai, toh hume ya toh training instability milti hai (wild oscillations, non-convergence) ya mode collapse (generator limited variety produce karta hai).

The Minimax Game: Balance Kyun Matters Karta Hai

jahan:

• $D(x)$ discriminator ki probability hai ki $x$ real hai
• $G(z)$ noise $z$ se ek fake sample generate karta hai
• Discriminator $D$ maximize karne ki koshish karta hai $V$ ko (real vs fake sahi classify kare)
• Generator $G$ minimize karne ki koshish karta hai $V$ ko (discriminator ko fool kare)

Training Instability ko First Principles se Derive Karna

Step 1: Discriminator kya optimize karta hai?

Fixed $G$ ke liye, discriminator maximize karta hai: $V(D,G)={\mathbb{E}}_{x\sim p_{data}}[\log D(x)]+{\mathbb{E}}_{x\sim p_g}[\log (1-D(x))]$

jahan $p_g$ generator ki distribution hai.

$D(x)$ ke respect mein functional derivative lete hain aur zero set karte hain: $\frac{\delta V}{\delta D(x)}=\frac{p_{data}(x)}{D(x)}-\frac{p_g(x)}{1-D(x)}=0$

Optimal discriminator ke liye solve karte hain: $D^{\ast }(x)=\frac{p_{data}(x)}{p_{data}(x)+p_g(x)}$

Yeh step kyun? Hum dhoondh rahe hain ki kaunse discriminator outputs maximum classification accuracy dete hain jab dono distributions known hain. Notice karo ki agar $p_{data}(x)=p_g(x)$ har jagah ho (perfect generator), toh $D^{\ast }(x)=\frac{1}{2}$ har jagah hoga—discriminator random guessing tak reduce ho jaata hai.

Step 2: Generator ko kaunsa gradient milta hai?

Generator gradient descent se optimize karta hai: ${\mathbb{E}}_{z\sim p_z}[\log (1-D(G(z)))]$

Chain rule use karte hue, generator parameters $\theta$ ke w.r.t. gradient hai: ${\nabla }_{\theta }{\mathbb{E}}_z[\log (1-D(G(z)))]={\mathbb{E}}_z\left[\frac{-1}{1-D(G(z))}\cdot {\nabla }_{x}D(G(z))\cdot {\nabla }_{\theta }G(z)\right]$

Yeh step kyun? Hume generator parameters $\theta$ update karne ke liye gradient chahiye. Notice karo ki do factors hain jo discriminator par depend karte hain: scalar $\frac{-1}{1-D(G(z))}$ aur term ${\nabla }_{x}D(G(z))$ (discriminator ka output kitna change hota hai jab hum sample ko move karte hain).

Step 3: Vanishing gradient problem (corrected)

Jab discriminator near-perfect hota hai, $D(G(z))\approx 0$ fake samples ke liye. Dono factors examine karte hain:

1. Scalar factor: $\frac{-1}{1-D(G(z))}\approx \frac{-1}{1-0}=-1$ — yeh ek constant hai, small nahi, isliye yeh akele vanishing gradient cause nahi karta.

2. Asli culprit hai ${\nabla }_{x}D(G(z))$. Ek confident, well-trained discriminator ka output saturated hota hai: woh $0$ (fake) ya $1$ (real) ke bahut kareeb values output karta hai aur beech mein flat rehta hai. Flat output ka matlab hai uska input-gradient ${\nabla }_{x}D\to 0$. Toh bhale hi scalar $-1$ hai, poora product

$\underset{\text{constant}}{\underbrace{(-1)}}\cdot \underset{\to 0}{\underbrace{{\nabla }_{x}D(G(z))}}\cdot {\nabla }_{\theta }G(z)\to 0$

vanish ho jaata hai. Generator ko almost koi learning signal nahi milta.

Yeh step kyun? Yeh ek common misconception correct karta hai: gradient isliye vanish nahi hota kyunki $\frac{-1}{1-D}$ shrink hota hai (yeh constant $-1$ ke paas jaata hai). Yeh isliye vanish hota hai kyunki ek bahut confident discriminator saturate kar jaata hai, jisse uska input-derivative ${\nabla }_{x}D\to 0$ ho jaata hai. Jab $D$ bahut zyada jeet jaata hai, $G$ improve karna band kar deta hai.

${\max }_G{\mathbb{E}}_{z\sim p_z}[\log D(G(z))]$

Gradient ban jaata hai: ${\nabla }_{\theta }{\mathbb{E}}_z[\log D(G(z))]={\mathbb{E}}_z\left[\frac{1}{D(G(z))}\cdot {\nabla }_{x}D(G(z))\cdot {\nabla }_{\theta }G(z)\right]$

Jab $D(G(z))\approx 0$ hota hai, scalar prefactor $\frac{1}{D(G(z))}$ bada ho jaata hai ($1/0.001=1000$). Yeh bada prefactor (chhote) saturated ${\nabla }_{x}D$ ko amplify karta hai, us learning signal ko rescue karta hai jo original loss ne khoi thi. Yeh training ke early stages mein stronger signal provide karta hai.

Yeh kyun kaam karta hai: Humne problem ko invert kar diya—"probability of being fake" minimize karne ki jagah, hum "probability of being real" maximize karte hain. Dono losses ke same fixed points hain lekin jab $D$ confident hota hai toh gradient magnitudes bahut different hoti hain.

Mode Collapse: The Copy-Paste Problem

Mode Collapse ko Game Dynamics se Derive Karna

Step 1: Generator ka incentive structure

Iteration $t$ par, maan lo generator ne sample $x_1$ produce karna seekh liya hai jo $D$ ko fool karta hai. $x_1$ produce karne ke liye generator ka loss hai: $L_G(x_1)=-\log D(x_1)$

Agar $D(x_1)=0.9$ ho (discriminator sochta hai yeh real hai), toh $L_G=-\log (0.9)\approx 0.105$ (low loss, accha!).

Step 2: Agar generator variety try kare toh kya hoga?

Ab maan lo generator true distribution se ek alag mode $x_2$ try karta hai. Lekin $D$ ne pehle $G$ ko $x_2$ type samples produce karte nahi dekha, isliye $D(x_2)=0.3$ (discriminator suspicious hai).

$L_G(x_2)=-\log (0.3)\approx 1.204$

Loss bahut zyada hai! Generator ko naye modes try karne par penalty milti hai.

Yeh step kyun? Generator ka objective sirf abhi $D$ ko fool karne ki parwah karta hai, full distribution cover karne ki nahi.

Step 3: Discriminator sirf current samples ko punish kar sakta hai

Discriminator update hota hai based on: $L_D=-{\mathbb{E}}_{x\sim p_{data}}[\log D(x)]-{\mathbb{E}}_z[\log (1-D(G(z)))]$

Agar $G$ kabhi mode $x_2$ se samples produce nahi karta, toh discriminator kabhi us mode ko check karna nahi seekhta. Generator ke paas use explore karne ka koi incentive nahi hai.

Yeh step kyun? Yahi mode collapse ka core hai—$G$ exploit karta hai $D$ ki unseen modes ke baare mein blindness ko.

Iteration 1-100: Generator digit "1" produce karna seekhta hai (generate karne mein sabse aasaan, almost straight line). Discriminator seekhta hai ki "1" suspicious hai.

Iteration 101-200: Generator digit "7" produce karne par switch karta hai (yeh bhi simple hai). Discriminator ab "7" detect karne par focused hai, "1" ke baare mein bhool jaata hai.

Iteration 201-300: Generator wapas "1" par switch karta hai. Discriminator drift kar chuka hai, "1" ko ab nahi pehchaanta.

Result: Generator 2-3 digit types ke beech oscillate karta hai, 0-9 ki full distribution kabhi nahi seekhta. Yeh mode collapse hai.

Yeh kyun hota hai: Generator ne paaya ki kuch modes ke beech switch karna sabhi modes cover karne se aasaan hai, aur discriminator saare modes ko simultaneously "remember" nahi kar sakta.

Solution: WGAN ek critic $f_w(x)$ use karta hai jo real number output karta hai (no sigmoid), aur Wasserstein distance optimize karta hai:

$W(p_{data},p_g)={\sup }_{\Vert f{\Vert }_L\le 1}{\mathbb{E}}_{x\sim p_{data}}[f(x)]-{\mathbb{E}}_{x\sim p_g}[f(x)]$

Critic constrained hota hai 1-Lipschitz continuous rehne ke liye (gradient clipping ya gradient penalty).

Generator loss ban jaata hai: $L_G=-{\mathbb{E}}_z[f_w(G(z))]$

Yeh kyun kaam karta hai:

1. Critic har jagah non-vanishing gradients provide karta hai (no sigmoid saturation)
2. Wasserstein distance generator distribution ko data distribution ki taraf saare modes mein "pull" karta hai, sirf currently visible modes mein nahi
3. Training zyada stable hoti hai kyunki objective sample quality ke saath correlate karta hai

Detailed steps:

1. Critic update (generator update ke per multiple steps): $w\leftarrow w+\alpha \cdot {\nabla }_w\left[\frac{1}{m}{\sum }_{i=1}^m{f}_w(x^{(i)})-\frac{1}{m}{\sum }_{i=1}^m{f}_w(G(z^{(i)}))\right]$ Phir clip karo: $w\leftarrow \text{clip}(w,-c,c)$ ya gradient penalty apply karo

2. Generator update: $\theta \leftarrow \theta -\alpha \cdot {\nabla }_{\theta }\left[-\frac{1}{m}{\sum }_{i=1}^m{f}_w(G(z^{(i)}))\right]$

Yeh steps kyun: Hum alternating kar rahe hain critic ko distinguishing mein better banane (step 1) aur generator ko fooling mein better banane (step 2) ke beech, lekin poore time continuous gradients ke saath.

Common Training Instabilities

Kyun sahi lagta hai: "Dono networks equally important hain, toh equal learning rates fair lagte hain."

Kyun galat hai: Discriminator ka task (already-generated samples par binary classification) generator ke task (noise se realistic samples create karna) se aasaan hai. Equal learning rates se, $D$ typically bahut jaldi seekhta hai aur bahut strong ban jaata hai.

Fix: $D$ ke liye $G$ se lower learning rate use karo, ya $G$ update ke per $D$ ko kam baar train karo. Common ratio: $D$ ko 1-3 $G$ updates per once train karo (ya TTUR use karo: Two Time-scale Update Rule with D aur G ke liye Adam mein different learning rates).

Kyun sahi lagta hai: "Jab $D(G(z))\to 0$ hota hai, toh surely gradient mein koi term zero tak shrink ho jaata hai aur learning ko kill kar deta hai."

Kyun galat hai: Scalar $\frac{-1}{1-D(G(z))}\to -1$ ek constant hai, small nahi. Asli cause yeh hai ki bahut confident discriminator saturate kar jaata hai, isliye uska input-gradient ${\nabla }_{x}D(G(z))\to 0$ ho jaata hai. Product vanish hota hai saturated derivative ki wajah se, scalar prefactor ki wajah se nahi.

Fix: Non-saturating loss $\log D(G(z))$ use karo (jiski prefactor $\frac{1}{D}$ blow up hoti hai chhote ${\nabla }_{x}D$ ko counteract karne ke liye), ya WGAN par switch karo jiska critic kabhi saturate nahi karta.

Kyun sahi lagta hai: "Batch norm doosre deep networks mein training stabilize karta hai, toh GANs mein bhi help karna chahiye."

Kyun galat hai: Batch norm batch mein samples ke beech dependencies create karta hai. Discriminator mein, yeh use fake batches (saare fakes BN statistics ke through correlated hain) vs real batches detect karne deta hai.

Fix:

• Virtual Batch Normalization (VBN) use karo: har sample ko ek fixed reference batch ki statistics use karke normalize karo
• Spectral Normalization use karo: batch dependencies ke bina layers ki Lipschitz constant control karta hai
• Generator mein Layer Normalization ya Instance Normalization use karo

Advanced Stabilization Techniques

Layer $l$ mein weight matrix $W$ ke liye, uski spectral norm (largest singular value) se divide karo: $\bar{W}=\frac{W}{\sigma (W)}$

jahan $\sigma (W)={\max }_{\Vert h\Vert =1}\Vert Wh{\Vert }_2$ spectral norm hai.

Yeh kyun kaam karta hai: Yeh har layer ki Lipschitz constant ko at most 1 hone ke liye constrain karta hai: $\Vert f_W(x)-f_W(y)\Vert \le \Vert x-y\Vert$

Discriminator mein extremely sharp transitions nahi ho sakti, jo mode collapse prevent karta hai aur training stabilize karta hai.

Efficiently compute kaise karein: Power iteration method use karo:

1. Random vector $v$ initialize karo
2. Iterate karo: $u=Wv/\Vert Wv\Vert$, $v=W^{T}u/\Vert W^{T}u\Vert$
3. Convergence ke baad: $\sigma (W)\approx u^{T}Wv$

Yeh step kyun: Exact SVD compute karna $O(n^3)$ hai; power iteration hume $O(n^2)$ per iteration mein $\sigma (W)$ deta hai, aur hume per training step sirf 1-2 iterations chahiye.

Weight clipping ki jagah, ek penalty term add karo: $\lambda {\mathbb{E}}_{\hat{x}}[(\Vert {\nabla }_{\hat{x}}{f}_w(\hat{x}){\Vert }_2-1{)}^2]$

jahan $\hat{x}=ϵx+(1-ϵ)G(z)$ with $ϵ\sim U[0,1]$ (real aur fake samples ke beech lines par points).

Yeh kyun kaam karta hai: Wasserstein distance require karta hai ki critic 1-Lipschitz ho. Yeh penalty enforce karta hai ki gradient norm 1 ke kareeb ho, jo Lipschitz constraint ka tightest form hai.

Derivation:

1. Ek 1-Lipschitz function satisfy karta hai $|f(x)-f(y)|\le \Vert x-y\Vert$
2. Differentiable $f$ ke liye, iska matlab hai $\Vert \nabla f(x)\Vert \le 1$ har jagah
3. Optimal critic mein $\Vert \nabla f(x)\Vert =1$ almost everywhere hota hai
4. Isliye hum norm = 1 se deviation ko penalize karte hain

Interpolated points kyun: Real aur fake samples ke beech ka region woh jagah hai jahan critic ko sabse sharply distinguish karna hota hai, isliye wahan hum constraint enforce karte hain.

Problem: High-resolution GANs (1024×1024) ko directly train karna unstable hota hai—early training ke liye bahut zyada detail hoti hai.

Solution: Low resolution (4×4) se start karo, gradually layers add karke resolution badhaao (8×8, 16×16, ..., 1024×1024). Har resolution stable hone tak train karta hai phir grow karta hai.

Implementation:

1. Start Generator outputs 4×4, Discriminator inputs 4×4
2. Layers add karo: $\alpha \in [0,1]$ use karke naye layers smoothly fade in karo: $\text{output}=\alpha \cdot \text{new\_layer}(x)+(1-\alpha )\cdot \text{upsample}(\text{old\_output})$
3. $\alpha$ ko 0 se 1 tak kaafi hazaar iterations mein increase karo
4. $\alpha =1$ par stabilize karo, phir next resolution layer add karo

Yeh kyun kaam karta hai: Low-resolution images mein kam information hoti hai, isliye GAN pehle broad structure seekhta hai (shape, layout) phir details (texture, fine edges). Yeh hierarchical learning sabhi scales simultaneously seekhne se zyada stable hai.

Monitoring aur GAN Training Diagnose Karna

Q: Kya signs hain ki discriminator bahut strong hai? A: Generator loss high aur constant rehti hai, $D(G(z))\approx 0$ saare generated samples ke liye, generated samples visually improve nahi hote

Q: Mode collapse ke kya signs hain? A: Generated samples mein low diversity (e.g., saare faces ka same pose/expression hai), generator loss oscillate karta hai lekin samples variety mein improve nahi hote, Inception Score ya FID suddenly drop ho jaata hai

Q: Tum oscillation aur convergence mein distinguish kaise kar sakte ho? A: Time ke saath loss curves plot karo—convergence mein stabilizing losses dikhti hain, oscillation mein periodic swings dikhte hain. Time ke saath sample diversity check karo—oscillation often matlab hota hai samples kuch types ke beech cycle karte hain.

Connections

• 4.5.01-Generative-Adversarial-Networks-fundamentals - Core GAN setup aur original objective
• 4.5.07-Wasserstein-GAN-and-improved-training - Instability ke liye WGAN solution
• 4.5.09-StyleGAN-and-progressive-growing - High-res stability ke liye progressive training
• 4.5.10-Conditional-GANs-and-control - Conditional generation mein mode collapse
• 3.4.05-Optimization-challenges-in-deep-learning - General optimization issues jo GANs se relevant hain
• 4.3.06-Batch-normalization-and-alternatives - Kyun batch norm GAN issues cause karta hai

Flashcards

#flashcards/ai-ml

Socho tumhare do dost ek game khel rahe hain. Ek dost (the Gener

Concept Map