Generative Adversarial Networks (GAN) framework

Overview

Generative Adversarial Networks (GANs) are a framework for training generative models through an adversarial process between two neural networks: a Generator that creates fake samples and a Discriminator that distinguishes real from fake. This competitive training regime produces highly realistic synthetic data.

The key insight: we don't need to explicitly specify what "realistic" means. The discriminator learns this implicitly from data, and competition forces the generator to match it.

The Two-Player Game Framework

1. Generator $G(z; \theta_g)$: Maps random noise $z \sim p_z(z)$ to data space, producing fake samples $G(z)$
2. Discriminator $D(x; \theta_d)$: Maps input $x$ to probability $[0,1]$ that $x$ is real (not generated)

The game: $G$ tries to maximize $D$'s error rate, while $D$ tries to minimize it.

The Objective Function: Deriving From First Principles

Starting Point: What do we want?

• $D$ should output 1 for real data, 0 for fake data
• $G$ should fool $D$ into outputing 1 for fake data

Step 1: Express $D$'s goal using probabilities

For real data $x \sim p_{data}(x)$, we want $D(x) \approx 1$. Using log probability (maximizes when $D(x) = 1$): $\mathbb{E}_{x \sim p_{data}}[\log D(x)]$

Why log? Logarithm converts probabilities to log-likelihoods, standard in maximum likelihood estimation. It also provides numerical stability and treats0.99→1.0 as less important than 0.01→0.1.

Step 2: Express $D$'s goal for fake data

For fake data $G(z)$ where $z \sim p_z(z)$, we want $D(G(z)) \approx 0$, equivalently $1 - D(G(z)) \approx 1$: $\mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]$

Step 3: Combine both objectives

$D$ wants to maximize both, so: $\max_D \left[\mathbb{E}_{x \sim p_{data}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]\right]$

Step 4: What does $G$ want?

$G$ wants to minimize $D$'s success, i.e., minimize the objective that $D$ maximizes: $\min_G \max_D \left[\mathbb{E}_{x \sim p_{data}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1 - D(G(z)))]\right]$

The training objective is: $\min_G \max_D V(G,D)$

This is a minimax two-player game. At equilibrium, $G$ generates samples indistinguishable from real data, and $D$ outputs $0.5$ everywhere (maximum uncertainty).

Training Algorithm: Alternating Optimization

Since we cannot solve the minimax problem directly, we use alternating gradient descent:

Why alternating? If we trained both simultaneously with joint gradient descent, the objectives conflict—improving one worsens the other's gradient signal. Alternating allows each network to adapt to the other's current state.

For Discriminator ($k$ steps):

Maximize $V(G,D)$ w.r.t. $\theta_d$: $\nabla_{\theta_d} V(G,D) = \mathbb{E}_{x \sim p_{data}}\left[\nabla_{\theta_d} \log D(x)\right] + \mathbb{E}_{z \sim p_z}\left[\nabla_{\theta_d} \log(1 - D(G(z)))\right]$

Practical computation:

1. Sample minibatch $\{x^{(1)}, \ldots, x^{(m)}\}$ from real data
2. Sample minibatch $\{z^{(1)}, \ldots, z^{(m)}\}$ from noise $p_z$
3. Generate fakes: $\{\tilde{x}^{(1)}, \ldots, \tilde{x}^{(m)}\}$ where $\tilde{x}^{(i)} = G(z^{(i)})$
4. Gradient ascent update: $\theta_d \leftarrow \theta_d + \alpha \cdot \frac{1}{m}\sum_{i=1}^{m}\left[\nabla_{\theta_d}\log D(x^{(i)}) + \nabla_{\theta_d}\log(1-D(\tilde{x}^{(i)}))\right]$

Why multiple $k$ discriminator steps? Early in training, $G$ is poor, so $D$ can easily classify. We want $D$ near optimal before updating $G$, otherwise $G$ receives weak gradient signal.

For Generator (1 step):

Minimize $V(G,D)$ w.r.t. $\theta_g$ — equivalently, maximize $-V(G,D)$: $\nabla_{\theta_g} (-V(G,D)) = -\mathbb{E}_{z \sim p_z}\left[\nabla_{\theta_g} \log(1-D(G(z)))\right]$

Key Issue: When $G$ is poor, $D(G(z)) \approx 0$, so $\log(1-D(G(z))) \approx 0$ → flat gradients (vanishing gradient problem).

Solution: Instead of minimizing $\log(1-D(G(z)))$, maximize $\log D(G(z))$: $\nabla_{\theta_g} \mathbb{E}_{z \sim p_z}[\log D(G(z)]$

Why does this work? Both objectives have the same optimal point (where $D(G(z)) = 0.5$), but $\log D(G(z))$ has stronger gradients when $D(G(z))$ is small. When $D$ confidently rejects ($D \approx 0$), $-\log D$ gives large gradient, providing strong learning signal.

Practical computation:

1. Sample minibatch $\{z^{(1)}, \ldots, z^{(m)}\}$ from $p_z$
2. Gradient ascent on $\log D(G(z))$: $\theta_g \leftarrow \theta_g + \alpha \cdot \frac{1}{m}\sum_{i=1}^{m}\nabla_{\theta_g}\log D(G(z^{(i)}))$

Iteration t=1000:

1. Discriminator update ($k=5$ steps):

   • Batch of64 real MNIST digits: $x^{(1)}, \ldots, x^{(64)}$
   • Sample $z^{(i)} \sim \mathcal{N}(0, I_{100})$, generate fakes $\tilde{x}^{(i)} = G(z^{(i)})$
   • Compute $D(x^{(i)})$ for reals, $D(\tilde{x}^{(i)})$ for fakes
   • Loss: $L_D = -\frac{1}{64}\sum_{i=1}^{64}[\log D(x^{(i)}) + \log(1-D(\tilde{x}^{(i)}))]$
   • Backprop, update $\theta_d$ to minimize $L_D$ (maximize $V$)
   • Repeat 5 times Why this step? After these updates, $D$ can better discriminate current $G$ outputs, providing meaningful feedback.

2. Generator update (1 step):

   • Sample fresh noise $z^{(1)}, \ldots, z^{(64)} \sim \mathcal{N}(0, I_{100})$
   • Generate $\tilde{x}^{(i)} = G(z^{(i)})$
   • Compute $D(\tilde{x}^{(i)})$
   • Loss: $L_G = -\frac{1}{64}\sum_{i=1}^{64}\log D(\tilde{x}^{(i)})$ (negative because we maximize)
   • Backprop through both $G$ and $D$ (but only update $\theta_g$)
   • Update $\theta_g$ to minimize $L_G$ Why backprop through D? The gradient $\nabla_{\theta_g} D(G(z))$ requires chain rule through $D$'s forward pass. We compute $\frac{\partial D}{\partial G(z)} \cdot \frac{\partial G(z)}{\partial \theta_g}$.

After many iterations: $G$ produces realistic digits, $D(x) \approx 0.5$ for both real and fake.

Epoch 10:

• $D(x_{real})$: 0.91 (confidently real)
• $D(G(z))$: 0.12 (confidently fake)
• Generated images: vague blob shapes

Why these values? $D$ has learned to discriminate easily because $G$ is still weak.

Epoch 50:

• $D(x_{real})$: 0.78
• $D(G(z))$: 0.31
• Generated images: digit-like structures, but blurry

Why values decreased? $G$ improved, making $D$'s job harder. Arms race in action.

Epoch 200 (near convergence):

• $D(x_{real})$: 0.58
• $D(G(z))$: 0.54
• Generated images: sharp, realistic digits

Why near 0.5? At equilibrium, $p_g = p_{data}$, so $D$ cannot distinguish, outputs maximum entropy (0.5).

Theoretical Properties

For fixed $G$, what is the optimal $D^*$?

The functional we maximize over $D$ is: $V(G,D) = \int_x p_{data}(x)\log D(x)\,dx + \int_z p_z(z)\log(1-D(G(z)))\,dz$

Change of variables: Let $y = G(z)$, with induced distribution $p_g(y)$: $V(G,D) = \int_x p_{data}(x)\log D(x)\,dx + \int_x p_g(x)\log(1-D(x))\,dx$

Combine integrals: $V(G,D) = \int_x \left[p_{data}(x)\log D(x) + p_g(x)\log(1-D(x))\right]dx$

Pointwise optimization: For each $x$, maximize: $f(Dx)) = p_{data}(x)\log D(x) + p_g(x)\log(1-D(x))$

Take derivative w.r.t. $D(x)$ and set to zero: $\frac{\partial f}{\partial D(x)} = \frac{p_{data}(x)}{D(x)} - \frac{p_g(x)}{1-D(x)} = 0$

Solve for $D(x)$: $p_{data}(x)(1-D(x)) = p_g(x)D(x)$ $p_{data}(x) = D(x)[p_{data}(x) + p_g(x)]$ $D^*(x) = \frac{p_{data}(x)}{p_{data}(x) + p_g(x)}$

Interpretation: Optimal $D$ outputs the Bayes-optimal probability that $x$ is real. When $p_g = p_{data}$, we get $D^* = 0.5$ everywhere.

At the global optimum where $p_g = p_{data}$: $V(G^*, D^*) = \mathbb{E}_{x \sim p_{data}}[\log 0.5] + \mathbb{E}_{x \sim p_{data}}[\log 0.5] = -2\log 2$

This connects to Jensen-Shannon Divergence: $JSD(p_{data} \| p_g) = \frac{1}{2}KL(p_{data} \| \frac{p_{data}+p_g}{2}) + \frac{1}{2}KL(p_g \| \frac{p_{data}+p_g}{2})$

The GAN objective can be rewritten as: $\min_G V(G, D^*_G) = -\log 4+ 2 \cdot JSD(p_{data} \| p_g)$

Why JSD? Minimizing the GAN objective is equivalent to minimizing the Jensen-Shannon divergence between real and generated distributions. JSD is symmetric and bounded $[0, \log 2]$, reaching 0 only when distributions are identical.

Common Training Challenges

Why it feels right: From $G$'s perspective, if $G(z)$ fools $D$, that seems like success. Why explore other modes?

The problem: $G$ maps all noise vectors to a single output (or small subset). Example: when generating faces, all outputs are the same face with slight variations. The generator "colapses" to a few modes of $p_{data}$ instead of covering all modes.

Root cause: The minimax objective only requires $G$ to fool $D$ on average, not to cover the full data distribution. If $D$ is weak in detecting repetition, $G$ exploits this.

Detection: Monitor diversity of generated samples. Compute metrics like Inception Score or measure if multiple different $z$ values produce nearly identical $G(z)$.

Fix strategies:

• Minibatch discrimination: Let $D$ see multiple samples simultaneously, detect lack of diversity
• Unrolled GANs: Update $G$ considering future $D$ updates, discouraging exploitation
• Multiple GANs: Train ensemble, each captures different modes

Why it feels right: In supervised learning, loss typically decreases monotonically. We expect similar behavior.

The problem: $G$ and $D$ losses oscillate wildly, never stabilizing. One network's improvement is the other's degradation. The system never reaches equilibrium.

Root cause: GANs don't optimize a single loss function—they're solving a minimax game. Game dynamics can cycle rather than converge. The gradient field has rotational components, not purely conservative.

Analogy: Think of $(G, D)$ as coordinates. Standard optimization descends a valley toward a minimum. GAN optimization is like two magnets repelling—trajectories can orbit indefinitely.

Detection: Plot $D$ loss and $G$ loss over time. Look for periodic oscillations rather than monotonic trends.

Fix strategies:

• Careful learning rates: Lower rates reduce oscillation amplitude
• Gradient penalty (WGAN-GP): Regularize $D$ to have bounded gradients, stabilizes training
• Spectral normalization: Constrain $D$'s Lipschitz constant, smooth gradient flow
• Two-timescale update rule (TUR): Different learning rates for $G$ and $D$

Why it feels right: More training usually helps in supervised learning.

The problem: When $D$ becomes too strong quickly, it perfectly separates real from fake ($D(x_{real}) \approx 1$, $D(G(z)) \approx 0$). Then $\log(1-D(G(z))) \approx 0$, giving $G$ near-zero gradients. $G$ stops learning—the arms race stalls.

Why mathematically: $\frac{\partial}{\partial \theta_g} \log(1-D(G(z)))$ involves $\frac{\partial D}{\partial G(z)}$. When $D(G(z)) \approx 0$, small changes in $G(z)$ don't change $D$'s output (it's saturated at 0), so gradient vanishes.

Fix: Use the non-saturating objective $\max_G \mathbb{E}_z[\log D(G(z))]$ instead of $\min_G \mathbb{E}_z[\log(1-D(G(z))]$. This gives strong gradients even when $D$ is confident.

Verification: When $D(G(z)) = 0.01$:

• Original: $\nabla \log(1-0.01) = \nabla(-0.01005) \approx -0.01$ (tiny)
• Non-saturating: $\nabla \log(0.01) = \nabla(-4.605) \approx 100 \times$ larger gradient

Architectural Considerations

1. Replace pooling with strided convolutions (in $D$) and fractional-strided convolutions (in $G$)

   • Why? Pooling loses spatial information deterministically. Strided convs learn downsampling, giving gradients throughout.

2. Use batch normalization in both $G$ and $D$

   • Why? Normalizes layer inputs, prevents mode collapse by ensuring activations don't saturate
   • Exception: Don't use in $G$'s output layer or $D$'s input layer (preserve raw data/generation scale)

3. Remove fully connected hidden layers for deeper architectures

   • Why? Fully connected layers have huge parameter counts, harder to train. All-convolutional uses spatial structure.

4. Use ReLU in $G$ (except output: tanh), LeakyReLU in $D$

   • Why? ReLU helps gradients flow in $G$. LeakyReLU in $D$ prevents dead neurons (avoids zero gradient for negative inputs).

5. Generator output: Use tanh activation

   • Why? Matches normalized input range $[-1, 1]$ if data is scaled to this range. Bounded output stabilizes training.

Connections to Other Concepts

Related Topics:

• Variational Autoencoders (VAE): Alternative generative model with explicit likelihood
• Maximum Likelihood Estimation: GANs implicitly minimize divergence, related to MLE
• Kullback-Leibler Divergence: KL vs. JS divergence in distribution matching
• Backpropagation: Required to compute $\nabla_{\theta_g} D(G(z))$
• Nash Equilibrium: GAN training seeks Nash equilibrium of two-player game
• Discriminative vs Generative Models: GANs are generative, using a discriminative component
• Convolutional Neural Networks: DCGAN uses conv architectures
• Mode Collapse Solutions: Techniques to improve GAN coverage

Upstream Dependencies:

• Neural Network Basics
• Gradient Descent Optimization
• Cross-Entropy Loss

Downstream Applications:

• Image Synthesis with GANs
• Style Transfer
• Data Augmentation with GANs
• Conditional GANs (cGAN)
• Waserstein GAN (WGAN)

[!recall- Explain It to a 12-Year-Old Imagine you want to learn to draw realistic butterflies, but you've never seen one—you only have a teacher who's seen tons of butterflies.

Here's what you do: You draw a butterfly (it starts as terrible scribbles). You show it to your teacher, and they say "That's fake, try again." You draw another, they say "Still fake." But you notice what they're looking at—the wings are wrong, the antenna are too thick. So you improve.

Meanwhile, your teacher is also getting better at spoting fakes. At first, your drawings are SO bad that it's easy for them. But as you get better, they have to look closer—"Hmm, the wing pattern isn't quite right..."

This goes back and forth: you (the Generator) get better at drawing, they (the Discriminator) get better at spotting mistakes. Eventually, your butterflies are so good that even your expert teacher can't tell if it's your drawing or a photo. That's when you've succeeded!

The "trick" is that you never actually saw real butterfly. You learned only by trying to fool your teacher. Your teacher's job forced you to learn what real butterflies look like, without anyone directly showing you.

Order matters: Train D first (let it learn current G), then improve G to beat the updated D.

Objective mnemonic: "Real-log-Real, Fake-log-NOT-fake"

• Real samples: $\log D(x)$ — want $D$ near 1
• Fake samples: $\log(1-D(G(z)))$ — want $D$ near 0 (so $1-D$ near 1)

Flashcards

#flashcards/ai-ml

What are the two neural networks in a GAN and what does each do? :: Generator $G$ creates fake samples from random noise $z$. Discriminator $D$ classifies inputs as real (from data) or fake (from $G$). They compete: $G$ tries to fool $D$, $D$ tries to detect fakes.

Write the full GAN minimax objective function :: $\min_G \max_D V(G,D) = \mathbb{E}_{x \sim p_{data}}[\log D(x)] + \mathbb{E}_{z \sim p_z}[\log(1-D(G(z)))]$. Discriminator maximizes (wants high probability on real, low on fake), Generator minimizes (wants to fool discriminator).

What is the non-convergence problem in GANs? :: $G$ and $D$ losses oscillate without stabilizing. Unlike supervised learning with single loss, GANs solve a minimax game—one's improvement is the other's degradation. Gradient field has rotational components, causing orbiting rather than convergence to fixed point.

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)