Foundations — Score-based generative models

This page assumes nothing. Before you can read the parent note, you need to recognise every squiggle it uses. We build them one at a time — plain words first, then a picture, then why the topic needs it. Do not skip: each item leans on the one before.

1. A data point $x$ and the space it lives in

The picture. Forget images for a second. Suppose each data point has only two numbers (say height and weight). Then every possible person is a single dot on a flat sheet of paper — one axis for height, one for weight. That flat sheet is the "space". Real datasets just have hundreds or thousands of axes instead of two; we draw two because paper is flat, but every idea below carries over unchanged.

Why the topic needs it. Everything else — probability, gradients, noise — is defined at points $x$ in this space. If you can't see "a data point is a dot on a landscape", nothing later will land.

2. Probability density $p(x)$ — the height of a landscape

The picture. Take the flat sheet from Section 1 and let it bulge upward wherever data crowds together. You now have a landscape: hills where realistic data sits, flat plains of near-zero height where nothing lives. The height of the terrain above point $x$ is $p(x)$.

The symbol $p(\cdot )$ just means "the height function". $p_{\text{data}}(x)$ means "the true landscape that produced our dataset" — the one we wish we knew.

Why the topic needs it. Generating a realistic sample means "land on a tall hill, not on the empty plains". So the entire game is about this landscape's shape.

3. The gradient ${\nabla }_x$ — the "which way is uphill?" arrow

Here is the first piece of real notation, and it is the heart of the whole topic. We introduce it slowly.

The picture. Stand on your landscape at some point. Look around: one compass direction rises fastest. Draw an arrow pointing that way, and make it long if the slope is steep, short if it's gentle. Do this at every point and you get a whole field of arrows carpeting the landscape. In 2-D each arrow has 2 numbers (one per axis); in $784$-D it has $784$ numbers.

Why the topic needs it. The parent note's star object, the score, is literally a gradient. If you own "gradient = uphill arrow", the score costs you nothing.

4. The logarithm $\log$ — a friendly re-scaler

The picture. Take the landscape of Section 2 and re-plot its height on a log scale. The trillion-fold hill is now just, say, $27$ units tall instead of $10^{12}$. The locations of hills and valleys don't move — only the vertical stretching changes. So "uphill on $p$" and "uphill on $\log p$" point the same way; the log just tames the numbers.

Why the topic needs it. The score is ${\nabla }_x\log p(x)$, not ${\nabla }_{x}p(x)$. Section 6 shows why the log is the trick that deletes the intractable constant.

5. Expectation $\mathbb{E}$ and the Gaussian $\mathcal{N}$ — averaging and the bell

Two symbols travel together throughout the parent note.

The picture. In 2-D, $\mathcal{N}(0,I)$ is a round fuzzy cloud of dots centred at the origin, dense in the middle and thinning outward. $\sigma$ controls how big the cloud is: small $\sigma$ = tight speck, large $\sigma$ = a cloud that fills the whole sheet.

Why the topic needs it. The trick that makes score learning possible (denoising score matching) is: take a clean data point, kick it by a random draw $ϵ\sim \mathcal{N}(0,I)$, and learn to undo the kick. No Gaussian, no denoising trick.

6. The partition function $Z$ — and why the score erases it

The one-line payoff. Watch the log and gradient team up:

The log turned the quotient into a subtraction (Section 4); $Z$ doesn't depend on $x$, so its gradient (its "uphill arrow", Section 3) is zero and it falls away.

Why the topic needs it. This is the whole reason score-based models exist: the score of the landscape is knowable even when the landscape's total volume $Z$ is not.

7. The score $s(x)$ — assembling the pieces

That single line is the entire parent topic in one object. Everything after it — score matching, noise levels, Langevin sampling — is how we estimate this arrow field and how we walk along it.

8. Two named ideas you'll meet downstream

Prerequisite map

See also the sibling topics that use these same bricks: Diffusion Models, Energy-Based Models, VAE, GAN.

Equipment checklist

Cover the right side and answer aloud. If any stalls you, re-read that section before opening the parent note.