Foundations — Generative Adversarial Networks (GAN) framework

Before you can read a single line of the parent note, you must be fluent in a small alphabet of symbols. This page introduces each one from absolute zero — plain words first, then the picture it stands for, then why the topic cannot live without it. Read top to bottom: every symbol is built only from symbols already explained above it.

1. The two worlds: data space and noise space

Everything in a GAN lives in one of two "spaces". A space here just means "the set of all possible things of one kind".

• Data space — the world of things we want to make. If we make images, a point in data space is one whole image.
• Noise space — a world of pure randomness, our raw material. A point here is just a list of random numbers.

Look at the figure: on the left is a blob of noise dots (structureless); on the right is the data blob (curved, structured — real images cluster on a thin curved region, not everywhere). The Generator's whole job is the red arrow: carry a noise point on the left onto the data region on the right.

Why the topic needs two spaces: we cannot ask a network to "imagine from nothing" — a computer needs some input. So we feed it randomness and ask it to transform randomness into data. Two spaces, one bridge.

2. Vectors, and the symbol $x$

Every image is stored as a long list of numbers (each pixel's brightness). A list of numbers is called a vector.

• $x$ (letter x) = a point in data space, i.e. one real sample. Picture: one actual photo from our dataset.
• $z$ (letter z) = a point in noise space, i.e. one random seed. Picture: a short list like $[0.4,-1.2,\dots ]$ of length 100.

Why: the parent note writes $z\in {\mathbb{R}}^{100}$ and output $28\times 28\times 1$. Now you know that just means "the input is a list of 100 random numbers, the output is an image-shaped list of 784 numbers".

3. Probability distributions: $p_{data}$, $p_z$, $p_g$

Not every point in data space is equally likely to be a real photo. A photo of a face is common; a photo of pure static is not. A probability distribution is a rule that tells you how likely each point is.

Three distributions appear in the parent note — memorise their pictures:

In the figure the black curve is $p_{data}$ (where real things live) and the red curve is $p_g$ (where fakes live). Early in training they sit apart. The entire goal of a GAN, said in one sentence: push the red curve until it lands exactly on top of the black curve. When $p_g=p_{data}$, the fakes are statistically indistinguishable from real data.

Why the topic needs distributions: "make realistic images" is vague. "Make $p_g$ equal $p_{data}$" is a precise, checkable target — and it is what the whole minimax game secretly optimises. See Discriminative vs Generative Models for why generative models care about $p_{data}$ at all.

4. Sampling and the symbol $\sim$

Picture: a hand dipping into the blob and picking one dot, favouring crowded spots.

Why: we never have the formula for $p_{data}$ — only a pile of examples we can sample. GANs are built to work from samples alone, never needing the formula. This is their superpower.

5. The two networks as functions: $G$ and $D$

A function is a machine: put something in, get something out. A neural network (see Neural Network Basics) is a function with tunable knobs, so it can learn the right input→output rule.

Follow the figure left→right: a seed $z$ enters $G$, becomes a fake image $G(z)$; that fake, or a real image $x$, enters $D$; out comes one number on the red 0–1 dial.

Why one number in $[0,1]$? Because "is this real?" is a yes/no question, and the honest answer is a probability. $D(x)=0.9$ means "90% sure real". This is exactly a binary classifier — the tool that scores it is Cross-Entropy Loss.

6. The knobs: ${\theta }_g$ and ${\theta }_d$

Picture: a mixing board with thousands of sliders. Training nudges each slider a little. The tool that decides which way to nudge is Gradient Descent Optimization.

That is why the parent writes $G(z;{\theta }_g)$ — the semicolon means "$G$ takes input $z$, and its behaviour is controlled by knobs ${\theta }_g$".

7. Expectation: the symbol $\mathbb{E}$

Picture: draw a thousand samples, compute $f$ on each, take the mean. In practice we approximate it by a minibatch average $\frac{1}{m}{\sum }_{i=1}^{m}f(x^{(i)})$ — that is literally what the parent's training loops compute.

Why: the GAN objective must judge $D$ over all data, not one point. Expectation is the honest "score across the whole distribution".

8. The logarithm: why $\log$ appears everywhere

Look at the red curve in the figure. Two facts make $\log$ the perfect scoring tool:

1. It punishes confident mistakes brutally. If $D$ should say "real" ($D\to 1$, so we want $\log D$ big) but instead says $D=0.01$, then $\log (0.01)\approx -4.6$ — a huge penalty. Near $p=0$ the curve plunges to $-\infty$.
2. It barely rewards already-good answers. From $D=0.99$ to $D=1.0$ the curve is almost flat. So the network stops obsessing over cases it already nails and focuses its effort where it is wrong.

Why $\log$ and not just $D$ itself? Multiplying many probabilities gives tiny numbers a computer rounds to zero; $\log$ turns products into sums (stable), and it is the natural language of likelihood — the same reasoning behind Kullback-Leibler Divergence and Cross-Entropy Loss.

9. Putting the alphabet together: reading the value function

You can now decode the parent's central line without fear: $V(G,D)={\mathbb{E}}_{x\sim p_{data}}[\log D(x)]+{\mathbb{E}}_{z\sim p_z}[\log (1-D(G(z)))]$

Read it as plain English, symbol by symbol you now know:

• ${\mathbb{E}}_{x\sim p_{data}}[\log D(x)]$ — "on average over real examples, how loudly does $D$ correctly shout real?" ($D(x)\to 1$, so $\log D(x)\to 0$, the best possible).
• ${\mathbb{E}}_{z\sim p_z}[\log (1-D(G(z)))]$ — "on average over fakes, how loudly does $D$ correctly shout fake?" ($D(G(z))\to 0$, so $1-D(G(z))\to 1$, so $\log (\cdot )\to 0$, again best).

$D$ turns its knobs ${\theta }_d$ to make $V$ big; $G$ turns its knobs ${\theta }_g$ to make $V$ small. That opposition is the game, and its balance point is a Nash Equilibrium.

10. The minimax notation: ${\min }_G{\max }_D$

Picture: a saddle — a mountain pass shaped like a Pringle. Walk one direction and you're at a peak (that's $D$ climbing); walk the perpendicular direction and you're at a valley (that's $G$ descending). The equilibrium is the seat of the saddle where neither can improve alone. This is exactly the Nash Equilibrium idea.

Prerequisite map

Equipment checklist

Test yourself — reveal only after you have answered aloud.