Intuition The one core ideaA GAN is two networks playing a game: a generator invents fake images, a critic/discriminator judges them, and the generator improves by trying to fool the judge. DCGAN, WGAN and StyleGAN are three upgrades to this game — better building blocks , a better scoring rule , and better control knobs — and to read them you must first know what every symbol in the score, the picture, and the game actually means.
This page assumes you know nothing beyond basic arithmetic and what a picture is. Every symbol used by the parent note is built here, one at a time, each resting on the previous.
Definition Image as a grid of numbersA digital image is a grid of tiny squares called pixels . Each pixel holds numbers describing its colour. A colour image has 3 channels — Red, Green, Blue — so it is three stacked grids.
Look at the figure: on the left is a tiny 4×4 picture; on the right are its three number-grids peeled apart.
3 × 64 × 64 3 channels , each a 64-row by 64-column grid. That's 3 × 64 × 64 = 12288 Why the topic needs this: a generator's whole job is to output such a grid of numbers . When the parent writes "3 × 64 × 64
Definition Normalized pixel range[ − 1 , 1 ] 0 255 − 1 + 1 [ − 1 , 1 ] v − 1 ≤ v ≤ 1
[ − 1 , 1 ] tanh — a function whose output is always between − 1 1
Definition A vector and the symbolR n vector is just an ordered list of numbers, e.g. [ 0.2 , − 0.5 , 0.8 ] R R n n z ∈ R 100 z is a list of 100 real numbers."
The bold z this is a vector, not one number . The symbol ∈ "is an element of / belongs to" .
Intuition Picture: a vector is a point (or arrow) in spaceA list of 2 numbers is a point on a flat sheet. A list of 3 numbers is a point in a room. A list of 100 numbers is a point in a space too big to draw — but the idea is identical: one dot, its position fixed by its numbers.
∥ x ∥ ∥ ⋅ ∥ Euclidean norm : the ordinary straight-line length of a vector, by Pythagoras. For x = [ x 1 , x 2 , … , x n ] ∥ x ∥ = x 1 2 + x 2 2 + ⋯ + x n 2 . ∥ x 1 − x 2 ∥ straight-line distance between two points x 1 x 2 ∥ ⋅ ∥ not the largest-coordinate (L ∞ L 1 ∣ x ∣
Definition The latent vectorz p ( z ) z random numbers fed into the generator — the "seed" of a fake image. The space it lives in (R 100 latent space . Latent = "hidden": these numbers have no direct meaning yet; the generator learns to turn each seed into an image.
We must say where the random numbers come from . The standard choice is the standard normal : z ∼ N ( 0 , I ) z 0 1 p ( z ) z ∼ p ( z ) z ∼ N ( 0 , I ) Uniform ( − 1 , 1 )
Why the topic needs it: every GAN starts from z z how z
A function is a machine: put something in, get something out. We write G ( z ) z G G θ
Symbol
Plain meaning
Input → Output
G θ ( z ) Generator latent vector → fake image
D ( x ) Discriminator (classic GAN)image → number in [ 0 , 1 ]
f w ( x ) Critic (WGAN)image → any real number = "realness score"
Intuition Discriminator vs. critic — one pictureThe discriminator answers a yes/no question ("real? — 87% sure"), so its output is squeezed into [ 0 , 1 ] how much question ("how real, on an open scale?"), so its output can be any real number. That single change is the heart of WGAN.
L loss L function that takes the network's outputs and returns one scalar number measuring "how badly we are doing right now" (bigger = worse). It maps the whole batch of predictions to a single number so we have something concrete to push down . Each player has its own loss: the parent's L D L G L
θ w weights (and biases). θ whole bag of dials for the generator ; w L
α How big a nudge is the learning rate α w ← w − α ⋅ ∇ w L ∇ w L α ←
∇ ∇ ∇ w L "for each dial in w L It is a vector of slopes — a compass pointing in the direction that increases L downhill (make L
Intuition Why a gradient at all?The loss L local downhill arrow — the only affordable way to know which way to step. This is why the parent's whole WGAN complaint — "zero gradient → no learning signal " — is fatal: a flat landscape gives the compass nothing to point at.
Definition Activation functionAfter each layer of a network, we bend the numbers through a small fixed function so the network can represent curves, not just straight lines. These are activations .
ReLU = max ( 0 , x ) 0 generator .LeakyReLU : like ReLU but negatives get a small slope instead of a hard 0 discriminator to avoid "dying" neurons (dials that get stuck because their gradient is always 0 tanh : an S-curve squashing any input into ( − 1 , 1 ) output so images land exactly in the normalized range from §1.
Recall Why tanh and not ReLU at the output?Because pixels must span [ − 1 , 1 ] [ − 1 , 1 ]
Definition A distributionA distribution describes how likely each possible value is. Picture a cloud of dots: dense where values are common, sparse where they're rare. P r real images; P g generated images.
x ∼ P r ∼ is drawn from ". Read "x E x ∼ P r [ f ( x ) ] E expectation = the average value of f ( x ) x
Intuition Why clouds and not single images?A generator that only makes one good image is useless. We want the whole cloud P g whole cloud P r distance between two clouds . WGAN's genius is choosing a better way to measure that distance.
Definition Manifold — the "thin sheet" that clouds live onReal images occupy only a razor-thin sheet inside the huge 12288 manifold . Two thin sheets in a vast space usually don't overlap — which is precisely why the parent says JS divergence breaks early in training.
Now that G D E ∼ actual game the parent starts from.
Intuition Reading the two linesFirst line: "make D ( real ) log D and make D ( fake ) log ( 1 − D ) D ( G ( z )) up toward 1 adversarial game. The parent's key claim is that when the two clouds don't overlap, D D ( G ( z )) ≈ 0 L G
Definition Divergence, infimuminf sup divergence is a number saying "how different are these two clouds." Two operators appear:
inf infimum ) = the greatest lower bound ≈ "the smallest achievable value."sup supremum ) = the least upper bound ≈ "the largest achievable value."
They are careful versions of "minimum" and "maximum" that still work when the exact best value is only approached, never reached.
inf sup inf W all transport plans, take the cheapest one" — the smallest possible shovelling cost is the distance. The sup all slope-limited critics, take the one that most separates the two average scores." So inf sup select the single best plan / best critic that defines the number. The critic network f w reach that sup
Definition The Lipschitz norm∥ f ∥ L ∥ f ∥ L ≤ 1 ∥ f ∥ L Lipschitz constant of f steepest slope f , measured as
∥ f ∥ L = sup x 1 = x 2 ∥ x 1 − x 2 ∥ ∥ f ( x 1 ) − f ( x 2 ) ∥ . ∥ f ∥ L ≤ 1 f 1-Lipschitz ) means that ratio never exceeds 1 ∥ f ( x 1 ) − f ( x 2 ) ∥ ≤ ∥ x 1 − x 2 ∥ never steeper than 45° . WGAN forces this so the critic's score can't blow up to infinity, keeping the sup
Normalized range minus1 to plus1
Functions G and D and critic f
Gradient and learning rate
Divergence JS vs Wasserstein
Every arrow points toward the parent topic the main note : you need each box before that note's symbols make sense.
Cover the right side and answer aloud.
What does 3 × 64 × 64 3 colour channels, each a 64×64 grid of pixels — one RGB image.
What does z ∈ R 100 z
Where do the numbers in z They are drawn from a fixed prior, usually z ∼ N ( 0 , I ) z ∼ p ( z )
What is the symbol ∈ "is an element of / belongs to."
What does the norm ∥ x 1 − x 2 ∥ The Euclidean (straight-line) distance between points
x 1 and
x 2 :
∑ i ( x 1 , i − x 2 , i ) 2 .
Difference between D ( x ) f w ( x ) D [ 0 , 1 ] f w
What is the loss L A function mapping the network's outputs to one scalar "how-bad" number that training pushes down.
What do θ w The bags of tunable weights for the generator and the critic respectively.
What does ∇ w L For each weight in w L
Why subtract α ⋅ ∇ w L To move down the loss landscape; α
Write the vanilla discriminator objective. L D = E x ∼ P r [ log D ( x )] + E z ∼ p ( z ) [ log ( 1 − D ( G ( z )))]
What does tanh guarantee about its output? It always lies strictly between − 1 1
Why LeakyReLU in the discriminator? It keeps a small gradient for negative inputs, preventing "dead" neurons.
What does x ∼ P r x
What is E x ∼ P r [ f ( x )] The average of f ( x )
Why do real images live on a "manifold"? They fill only a thin sheet of the huge pixel space; random grids are static, not photos — so real/fake sheets rarely overlap.
Why does JS divergence give no gradient on disjoint clouds? It saturates at the constant log 2
How is W inf W = inf γ E ( x , y ) ∼ γ ∥ x − y ∥ inf
What is ∥ f ∥ L ∥ f ∥ L ≤ 1 The largest slope ratio ∥ f ( x 1 ) − f ( x 2 ) ∥/∥ x 1 − x 2 ∥ ≤ 1
