Visual walkthrough — Score-based generative models

We link back to the parent Score-based generative models and lean on Langevin Dynamics, Stochastic Differential Equations, and the cousin idea in Diffusion Models.

Step 1 — A cloud of points and the "uphill arrow"

WHY. Generative modelling means: invent new dots that look like they belong to this pile. One way is to know the height of the pile everywhere. But a cheaper way is to know, at any spot you stand, which way is uphill — the direction that leads toward thicker piles. Follow those arrows and you walk from empty desert into the crowd. That arrow field is the whole game.

PICTURE. Below, black dots are data. Stand anywhere; the red arrow points toward denser data. That red arrow is what we will call the score.

Step 2 — Why the log, and why the normalizer vanishes

WHY. Real densities are written $p(x)=\frac{\tilde{p}(x)}{Z}$.

• $\tilde{p}(x)$ — an unnormalized score of plausibility we can compute (e.g. $e^{-\text{energy}}$, see Energy-Based Models).
• $Z=\int \tilde{p}(x)dx$ — the partition function, the total volume under $\tilde{p}$. In high dimensions this integral is hopeless.

Watch what the log does:

${\nabla }_x\log p(x)={\nabla }_x[\underset{\text{depends on }x}{\underbrace{\log \tilde{p}(x)}}-\underset{\text{constant in }x}{\underbrace{\log Z}}]={\nabla }_x\log \tilde{p}(x).$

• $\log$ turned the division by $Z$ into a subtraction of $\log Z$.
• ${\nabla }_x$ of a constant is zero, so $\log Z$ evaporates.

PICTURE. Multiplying the whole pile by $1/Z$ rescales every height by the same factor — it never changes which way is uphill. The red uphill arrows are identical before and after normalizing.

Step 3 — The wall: we still don't know the true score

WHY / the naïve objective. Measure disagreement as squared arrow-difference, averaged over data:

$J(\theta )={\mathbb{E}}_{x\sim p_{\text{data}}}[\frac{1}{2}\Vert \underset{\text{our guessed arrow}}{\underbrace{s_{\theta }(x)}}-\underset{\text{true arrow — UNKNOWN}}{\underbrace{{\nabla }_x\log p_{\text{data}}(x)}}{\Vert }^2].$

• $s_{\theta }(x)$ — the network's arrow, tunable via weights $\theta$.
• $\Vert \cdot {\Vert }^2$ — squared length of the gap arrow; zero only when the two arrows match exactly.
• ${\mathbb{E}}_{x\sim p_{\text{data}}}$ — average over places where real data actually sits.

The wall: the second term is the very thing we don't have. We have samples from $p_{\text{data}}$, not its formula, so we cannot compute its gradient.

PICTURE. We can see dots (samples) but not the smooth height surface behind them. Asking for its gradient at a lonely dot is undefined — the surface is invisible.

Step 4 — The trick: blur the pile with Gaussian noise

WHY. A smooth cloud has a well-defined score everywhere — even in the deserts, because the blur leaks probability outward. And, as Step 5 shows, for Gaussian blur we know the score exactly.

The kick: $\tilde{x}=x+\sigma ϵ,ϵ\sim \mathcal{N}(0,I).$

• $x$ — a clean data point.
• $ϵ$ — a random push, each coordinate an independent standard bell-curve draw.
• $\sigma$ — how hard we push (the noise scale). Big $\sigma$ = heavy blur; $\sigma \to 0$ = original data.
• $\tilde{x}$ — the noisy point we actually feed the network.

PICTURE. Left: sharp dots. Right: the same dots each nudged by a red arrow $\sigma ϵ$, producing a soft haze whose height surface is smooth.

Step 5 — The exact score of one Gaussian blur

WHY this is computable. Start from the log of a Gaussian: $\log \mathcal{N}(\tilde{x};x,{\sigma }^{2}I)=-\frac{1}{2{\sigma }^2}\Vert \tilde{x}-x{\Vert }^2+\text{const}.$

• $\Vert \tilde{x}-x{\Vert }^2$ — squared distance from the fuzzy point to the true center.
• The minus sign — far from center means lower log-probability (a downhill bowl).
• $\text{const}$ — collects $\log Z$ and the $\sqrt{2\pi }$ factors; gradient kills it.

Now take the gradient with respect to $\tilde{x}$ (derivative of $-\frac{1}{2{\sigma }^2}\Vert \tilde{x}-x{\Vert }^2$ is $-\frac{1}{{\sigma }^2}(\tilde{x}-x)$):

${\nabla }_{\tilde{x}}\log q_{\sigma }(\tilde{x}\mid x)=-\frac{1}{{\sigma }^2}(\tilde{x}-x).$

Substitute the kick $\tilde{x}-x=\sigma ϵ$:

$\boxed{{\nabla }_{\tilde{x}}\log q_{\sigma }(\tilde{x}\mid x)=-\frac{\sigma ϵ}{{\sigma }^2}=-\frac{ϵ}{\sigma }}$

• $-\frac{ϵ}{\sigma }$ — the uphill arrow points opposite the noise you added, scaled by $1/\sigma$. It says: "to climb back to the clean point, undo the kick."

PICTURE. The noise $\sigma ϵ$ pushed the point out (black arrow); the score $-ϵ/\sigma$ points back in toward $x$ (red arrow) — exactly antiparallel.

Step 6 — From one point's score to the whole cloud's score

WHY (the denoising objective). Replace the unknown true score in Step 3 with the known per-point score. The new objective:

$J(\theta )={\mathbb{E}}_{x\sim p_{\text{data}}}{\mathbb{E}}_{ϵ\sim \mathcal{N}(0,I)}[\frac{1}{2}\Vert \underset{\text{network’s arrow}}{\underbrace{s_{\theta }(x+\sigma ϵ)}}+\underset{\text{easy target}}{\underbrace{\frac{ϵ}{\sigma }}}{\Vert }^2].$

• $x+\sigma ϵ$ — the noisy input the network sees.
• $+\frac{ϵ}{\sigma }$ — adding this equals subtracting the target $-ϵ/\sigma$; the bracket is zero when $s_{\theta }=-ϵ/\sigma$.
• Averaging over both $x$ and $ϵ$ — the key. For a fixed noisy point $\tilde{x}$, many different $(x,ϵ)$ pairs could have produced it. The squared error is minimized by the average of their targets, and that average is exactly ${\nabla }_{\tilde{x}}\log q_{\sigma }(\tilde{x})$ — the marginal score.

So: the best possible network output at $\tilde{x}$ is the average over all clean points that might have landed there, which is precisely the true smoothed score. The per-point noise is a proxy whose expectation is the truth.

PICTURE. One noisy point $\tilde{x}$ (center) could have come from several clean dots. Their individual back-arrows (grey) average to one red arrow — the marginal score the network converges to.

Step 7 — Edge cases: $\sigma$ too big, $\sigma$ too small

Case $\sigma \to 0$ (barely any noise). The cloud is razor-sharp, back on the true data manifold. But then the desert between modes stays empty: if you start sampling out there, the score is $\approx 0$ (flat, no signal) and you never find the data. Also the target $-ϵ/\sigma$ explodes, giving huge, unstable gradients.

Case $\sigma$ large (heavy blur). Now the whole space is gently sloped — great for connectivity, you always feel some uphill pull — but the fine structure of the data is smeared away, so samples come out blurry.

The fix (multi-scale). Use a ladder of noise levels ${\sigma }_1>{\sigma }_2>\cdots >{\sigma }_L$ and one noise-conditional network $s_{\theta }(x,\sigma )$ that is told which rung it is on: $J(\theta )={\sum }_{i=1}^L\underset{\text{weight}}{\underbrace{{\sigma }_i^2}}{\mathbb{E}}_x{\mathbb{E}}_{ϵ}[\frac{1}{2}\Vert s_{\theta }(x+{\sigma }_iϵ,{\sigma }_i)+\frac{ϵ}{{\sigma }_i}{\Vert }^2].$

• ${\sigma }_i^2$ weight — cancels the $1/{\sigma }_i^2$ blow-up in the target magnitude so every rung contributes comparably. Without it, large-$\sigma$ rungs are ignored.

PICTURE. Three panels: heavy blur (connected but coarse) → medium → light (sharp but isolated). The red arrow field grows sharper and more detailed as $\sigma$ shrinks.

Step 8 — Using the arrows: walk uphill with a wobble

WHY the jiggle. Pure uphill steps collapse onto the single highest peak (one image forever). The controlled noise (from Langevin Dynamics) lets you sample the whole distribution instead of only its summit. This is a discretization of the Langevin SDE $dx={\nabla }_x\log p(x)dt+\sqrt{2}dW$:

$x_{t+1}=x_t+\underset{\text{uphill drift}}{\underbrace{{\alpha }_i{s}_{\theta }(x_t,{\sigma }_i)}}+\underset{\text{exploration wobble}}{\underbrace{\sqrt{2{\alpha }_i}{z}_t}},z_t\sim \mathcal{N}(0,I).$

• ${\alpha }_i$ — step size; shrink it as $\sigma$ shrinks (finer moves near the data).
• $\sqrt{2{\alpha }_i}$ — the exact noise scale that makes the walk settle onto $p$ (detailed balance).
• Annealing: run this at ${\sigma }_1$ (coarse exploration), then ${\sigma }_2,\dots$ down to ${\sigma }_L$ (fine polish), carrying $x$ over between rungs.

PICTURE. A red trajectory zig-zags from an outer noisy start, guided by the arrow field, wobbling but steadily drifting into a data mode.

The one-picture summary

Everything on one canvas: clean data → blur into a smooth cloud whose per-point score is $-ϵ/\sigma$ → a network averages those to learn the true marginal arrows across a ladder of scales → annealed Langevin walks the red arrows from noise back to data.