Worked examples — Score-based generative models

This page lives under Score-Based Generative Models. Read that first for the definitions — here we do. We will take the score $s(x)={\nabla }_x\log p(x)$, the denoising target $-ϵ/\sigma$, and the Langevin update, and run them through every case a problem can throw at you: positive/negative gradients, the zero (peak) input, the degenerate $\sigma \to 0$ limit, the giant-$\sigma$ limit, a multi-dimensional case, a word problem, and an exam twist.

---

The scenario matrix

Before any example, here is the full space of cases this topic contains. Each worked example is tagged with the cell it covers.

Cell	What varies	Question it answers
A. Sign right	$x$ is to the right of the mode	Which way does the score point?
B. Sign left	$x$ is to the left of the mode	Does the arrow flip?
C. At the peak	$x$ = the mode exactly	What is the score when $p$ is maximal?
D. Small-$\sigma$ limit	$\sigma \to 0$	Why do scores blow up near clean data?
E. Large-$\sigma$ limit	$\sigma$ huge	Why does everything look like one big Gaussian?
F. Multi-dimensional	vector $x\in {\mathbb{R}}^d$	Score as a vector, not a number
G. Denoising target	given $\tilde{x},x,\sigma$	Compute $-ϵ/\sigma$ and check it equals the Gaussian score
H. Word problem	Langevin steps by hand	Does an actual dust speck move toward the mode?
I. Exam twist	mixture of two Gaussians	Where is the score zero and where does it point between modes?

Our worked-through model distribution for most cells is the humble 1-D Gaussian $p(x)=\mathcal{N}(x;\mu ,{\sigma }^2)=\frac{1}{\sqrt{2\pi {\sigma }^2}}\exp (-\frac{(x-\mu {)}^2}{2{\sigma }^2}).$

Where does that formula come from? $\log p(x)=-\frac{(x-\mu {)}^2}{2{\sigma }^2}+\text{const}$; differentiating $-\frac{(x-\mu {)}^2}{2{\sigma }^2}$ with respect to $x$ gives $-\frac{2(x-\mu )}{2{\sigma }^2}=-\frac{x-\mu }{{\sigma }^2}$. The constant (which hides the intractable $Z$) vanishes — exactly the magic from the parent note.

---

Cell A + B — the sign of the score (right vs left of the mode)

1. Write the score. $s(x)=-\frac{x-\mu }{{\sigma }^2}=-\frac{x-0}{1}=-x.$
   • Why this step? We plug $\mu =0,{\sigma }^2=1$ into the one formula. No integrals, no $Z$ — the log-gradient already killed the normalizer.
2. Evaluate right side. $s(3)=-3.$
   • Why? A negative score means "decrease $x$" — the arrow points left, back toward $0$.
3. Evaluate left side. $s(-3)=-(-3)=+3.$
   • Why? A positive score means "increase $x$" — the arrow points right, again toward $0$. The sign flips exactly as the side flips.

Figure caption (s02): The blue curve is $\log p(x)$, an upside-down parabola peaking at the dashed yellow line $x=0$. The red dot sits at $x=+3$ with a red arrow pointing left (that's $s(3)=-3$, "go downhill in $x$, uphill in $\log p$"). The green dot sits at $x=-3$ with a green arrow pointing right ($s(-3)=+3$). Look how both arrows aim at the peak — that is the entire message of cells A and B.

Verify: Both arrows point toward the mode $\Rightarrow$ Langevin will pull particles inward, which is what "follow the gradient uphill in $\log p$" must do. Magnitudes are equal ($3$) because the points are equidistant from $\mu$. ✓ (Cells A and B done.)

---

Cell C — the score at the peak

1. Plug in $x=\mu$. $s(0)=-\frac{0-0}{1}=0.$
   • Why this step? At the top of a smooth hill the slope is flat in every direction, so the gradient is the zero vector.
2. Interpret. A zero score is a fixed point of Langevin (ignoring the noise term). The deterministic push is zero; only the random kick $\sqrt{2\alpha }{z}_t$ (with $z_t\sim \mathcal{N}(0,1)$, defined at the top) moves you.
   • Why it matters: This is why samples cluster at modes but never freeze there — the noise term keeps them alive.

Verify: $-\frac{d}{dx}\left[\frac{x^2}{2}\right]{|}_{x=0}=-x{|}_{x=0}=0.$ The maximum of $\log p$ has zero derivative — first-derivative test confirms it. ✓ (Cell C.)

---

Cell D + E — the two limits of noise level $\sigma$

The denoising score at a noised point is $-\frac{\tilde{x}-x}{{\sigma }^2}$. Watch what $\sigma$ does to it.

1. Score formula for the noised Gaussian. $s=-\frac{\tilde{x}-x}{{\sigma }^2}=-\frac{0.1}{{\sigma }^2}.$
   • Why this step? $q_{\sigma }(\tilde{x}\mid x)=\mathcal{N}(\tilde{x};x,{\sigma }^2)$, so its score is displacement/variance, negated.
2. Small $\sigma =0.01$. $s=-\frac{0.1}{(0.01{)}^2}=-\frac{0.1}{0.0001}=-1000.$
   • Why? With tiny noise, being $0.1$ away from clean data is enormously improbable, so the log-density plummets steeply — a huge arrow yanks you back. This is cell D: scores explode near the data manifold. That is precisely why we need multiple noise levels.
3. Large $\sigma =10$. $s=-\frac{0.1}{100}=-\mathrm{0.001.}$
   • Why? With huge noise the distribution is a wide, flat blanket; $0.1$ off is nothing, so the arrow is nearly zero. This is cell E: at large $\sigma$ the landscape is smooth and easy to explore.

Figure caption (s03): Vertical axis is $|\text{score}|$ on a log scale; horizontal axis is the noise level $\sigma$. The blue curve $|s|=0.1/{\sigma }^2$ rockets up on the left (small $\sigma$, red dot — the huge arrow of cell D) and flattens toward the right (large $\sigma$, green dot — the whisper of cell E).

Verify: Ratio of magnitudes $=\frac{1000}{0.001}=10^6=(\frac{10}{0.01}{)}^2=({\sigma }_{\text{big}}/{\sigma }_{\text{small}}{)}^2$. Score magnitude scales as $1/{\sigma }^2$ — exactly why the $\lambda (\sigma )={\sigma }^2$ weighting in the parent note rescues balanced gradients. ✓ (Cells D and E.)

---

Cell F — the score is a vector in high dimensions

1. Vector form of the formula. $s(x)=-\frac{x-\mu }{{\sigma }^2}$ applied componentwise (because $\Sigma ={\sigma }^{2}I$ is diagonal — dimensions don't interact).
   • Why this step? For an isotropic Gaussian the log-density splits into a sum over coordinates, so the gradient splits too.
2. Displacement vector. $x-\mu =(3-1,0-(-2))=(2,2).$
3. Divide by variance, negate. $s=-\frac{1}{4}(2,2)=(-0.5,-0.5).$
   • Why? First component negative ⇒ push left (we're right of ${\mu }_1=1$). Second component negative ⇒ push down toward ${\mu }_2=-2$, since $0>-2$.

Verify: Both components point back toward $\mu =(1,-2)$: from $(3,0)$ the mean is down-left, and the score $(-0.5,-0.5)$ points down-left. Magnitude $\Vert s\Vert =\sqrt{0.5^2+0.5^2}=\sqrt{0.5}=\frac{1}{\sqrt{2}}\approx 0.707$. ✓ (Cell F.)

---

Cell G — denoising target equals the Gaussian score

1. Denoising target. $-\frac{ϵ}{\sigma }=-\frac{2}{0.5}=-4.$
   • Why this step? This is what the network is trained to output given $\tilde{x}$.
2. Gaussian score of $q_{\sigma }(\tilde{x}\mid x)$. $-\frac{\tilde{x}-x}{{\sigma }^2}=-\frac{6-5}{0.25}=-4.$
   • Why? This is the true score at the noised point. They agree because $\tilde{x}-x=\sigma ϵ$, hence $-\frac{\tilde{x}-x}{{\sigma }^2}=-\frac{\sigma ϵ}{{\sigma }^2}=-\frac{ϵ}{\sigma }$.

Verify: $-4=-4$. The identity $-\frac{ϵ}{\sigma }=-\frac{\tilde{x}-x}{{\sigma }^2}$ holds algebraically for any inputs, so denoising ≡ score learning. ✓ (Cell G — this is the whole justification of the DSM objective.)

---

Cell H — a word problem: run Langevin by hand

1. Step 1 drift. With $z_0=0$ the update is $x_1=x_0+\alpha s(x_0)=2+0.1\cdot (-2)=2-0.2=\mathrm{1.8.}$
   • Why this step? The score $s(x_0)=-x_0=-2$ says "go left"; scaled by $\alpha =0.1$ it moves the speck $0.2$ toward the mode. The $\sqrt{2\alpha }{z}_0$ term is $0$ because we chose $z_0=0$.
2. Step 2 drift. With $z_1=0$: $x_2=x_1+\alpha s(x_1)=1.8+0.1\cdot (-1.8)=1.8-0.18=\mathrm{1.62.}$
   • Why? Same rule with the new position. Each drift step multiplies the distance by $(1-\alpha )=0.9$, so it's a geometric crawl inward.
3. General pattern. After $t$ zero-kick steps $x_t=2\cdot (0.9{)}^t$.
   • Why it matters: Converges to $0$ (the mode) as $t\to \infty$ — exactly the "follow the score uphill" behaviour. In real sampling the kicks $z_t\sim \mathcal{N}(0,1)$ are nonzero and let the speck wander, so it samples the full Gaussian instead of collapsing onto $0$.

Verify: $x_1=2(0.9)=1.8$ ✓, $x_2=2(0.9{)}^2=2(0.81)=1.62$ ✓. Both between the start and the mode, monotonically decreasing. ✓ (Cell H.)

---

Cell I — exam twist: the score of a mixture

A quick tool we need first, because it appears everywhere in mixtures.

1. Symmetry argument. $p(x)$ is even about $x=0$ (swapping $x\to -x$ swaps the two bumps but leaves the sum unchanged), so $\log p$ is even, so its derivative — the score — is odd: $s(-x)=-s(x)$. An odd function passing through the origin must satisfy $s(0)=0.$
   • Why this step? Symmetry hands us the fixed point with no calculus at all.
2. Set up the score honestly. Write the two unnormalized bumps $g_{\pm }(x)=\exp (-\frac{(x\mp a{)}^2}{2{\sigma }^2})$, so $p(x)=\frac{1}{2\sqrt{2\pi {\sigma }^2}}(g_{+}(x)+g_{-}(x))$. The score is the log-derivative: $s(x)=\frac{d}{dx}\log (g_{+}(x)+g_{-}(x))=\frac{g_{+}^{\prime }(x)+g_{-}^{\prime }(x)}{g_{+}(x)+g_{-}(x)}.$
   • Why this step? $\log (\text{const}\cdot f)$ differentiates to $f^{\prime }/f$; the constant prefactor (our old friend $Z$) drops out.
3. Differentiate each bump. $g_{\pm }^{\prime }(x)=-\frac{x\mp a}{{\sigma }^2}{g}_{\pm }(x)$ (chain rule on the exponent). Substituting, $s(x)=\frac{-\frac{x-a}{{\sigma }^2}{g}_{+}(x)-\frac{x+a}{{\sigma }^2}{g}_{-}(x)}{g_{+}(x)+g_{-}(x)}.$
   • Why this step? Each bump's own score is $-(x\mp a)/{\sigma }^2$; the denominator $g_{+}+g_{-}$ is the density (up to the constant), so this is a weighted average of the two individual scores, weighted by how responsible each bump is for the point $x$.
4. Collapse the exponentials into $\tanh$. Divide top and bottom by $g_{+}(x)+g_{-}(x)$ and factor. The mean-of-two-scores rearranges to $s(x)=-\frac{x}{{\sigma }^2}\cdot {\sigma }^2\cdot \frac{1}{{\sigma }^2}+\frac{a}{{\sigma }^2}\cdot \frac{g_{+}(x)-g_{-}(x)}{g_{+}(x)+g_{-}(x)}.$ Now note $\frac{g_{+}(x)-g_{-}(x)}{g_{+}(x)+g_{-}(x)}=\frac{e^{ax/{\sigma }^2}-e^{-ax/{\sigma }^2}}{e^{ax/{\sigma }^2}+e^{-ax/{\sigma }^2}}=\tanh \left(\frac{ax}{{\sigma }^2}\right)$ (cancel the common factor $e^{-(x^2+a^2)/2{\sigma }^2}$ from both bumps, leaving exactly the $\tanh$ ratio). Taking the common case ${\sigma }^2=1$ this simplifies to the clean closed form $\boxed{s(x)=-x+a\tanh \left(ax\right)}({\sigma }^2=1).$
   • Why introduce $\tanh$ here? Because "difference of two exponentials over their sum" is the definition of $\tanh$ (see the definition box). It is the natural language for "which of two competing bumps wins", so it compresses the messy fraction into one interpretable S-curve.
5. Plug numbers ($a=2,{\sigma }^2=1$). At the centre $x=0$: $s(0)=-0+2\tanh (0)=0$ — the predicted fixed point. Just right of centre at $x=1$: $s(1)=-1+2\tanh (2)\approx -1+2(0.9640)=0.9281>0$ — the score points right, toward the near mode $+2$.
   • Why? A positive score at $0^{+}$ means the fixed point at $0$ is an unstable ridge, not a valley: the tiniest nudge sends the speck rolling to one mode. This is the reason single-scale Langevin gets stranded between modes, and why annealing from large $\sigma$ (where the two bumps merge into one broad hill) is essential.

Figure caption (s04): Blue is the two-humped density $p(x)$ (scaled up so you can see it); red is the score $s(x)=-x+2\tanh (2x)$. The red curve crosses zero at the dashed yellow line $x=0$ (the unstable ridge, yellow dot), is positive just to its right (green dot at $x=1$ — pushes toward $+2$) and negative just to its left. Watch how the red curve dives back down near each hump: those downward crossings at $x\approx \pm 2$ are the stable fixed points at the modes.

Verify: $s(0)=-0+2\tanh (0)=0$ ✓ (fixed point). $s(1)=-1+2\tanh (2)\approx 0.9281>0$ ✓ (repelled toward the near mode). By oddness $s(-1)\approx -0.9281$. ✓ (Cell I.)

---

Recall checkpoints

Recall Sign of the score right of the mode

Right of $\mu$ (so $x>\mu$), the Gaussian score $-(x-\mu )/{\sigma }^2$ is... ::: negative — it points left, back toward the mode.

Recall Score at the peak

The score exactly at the mode equals... ::: zero (flat top of the log-density hill).

Recall Why scores blow up at small sigma

As $\sigma \to 0$ the score magnitude scales like... ::: $1/{\sigma }^2$, so it explodes near clean data — motivating multiple noise scales.

Recall Denoising ↔ score identity

$-ϵ/\sigma$ equals the true score because... ::: $\tilde{x}-x=\sigma ϵ$, so $-(\tilde{x}-x)/{\sigma }^2=-ϵ/\sigma$.

Recall Mixture-of-two score

For $\frac{1}{2}\mathcal{N}(-a,{\sigma }^2)+\frac{1}{2}\mathcal{N}(a,{\sigma }^2)$ with ${\sigma }^2=1$, the score is... ::: $s(x)=-x+a\tanh (ax)$, zero at $x=0$ (an unstable ridge).

See also: Diffusion Models, Langevin Dynamics, Stochastic Differential Equations, Energy-Based Models, and the generative cousins VAE and GAN.