Denoising diffusion probabilistic models (DDPM): Visual walkthrough

This page has one job: to show you, from absolutely nothing, why you can jump straight from a clean image to a noisy image at any timestep in a single formula — the result the parent note calls the Closed-Form Forward Process:

$q (x_{t} ∣ x_{0}) = N (x_{t}; \overset{α}{ˉ}_{t} x_{0}, (1 - \overset{α}{ˉ}_{t}) I)$

We will earn every symbol. By the end you will see why the two square-root factors are exactly those and not anything else.

Before we start, four words you must never let pass undefined:

Definition The four primitive objects

$x_{0}$ — the clean data point (think: one photo, flattened into a long list of numbers, each between $- 1$ and $1$ ).
$ϵ_{t}$ — at each step $t$ we draw a fresh, independent blob of pure random noise, one number per pixel, each from the standard bell curve $N (0, 1)$ (average $0$ , spread $1$ ). We label these draws $ϵ_{1}, ϵ_{2}, \dots$ — different draws, never reused, and independent of each other. When only one draw is around we drop the subscript and just write $ϵ$ .
$β_{t}$ — a small positive number, the amount of noise we inject at step $t$ , with $0 < β_{t} < 1$ . The list $β_{1}, \dots, β_{T}$ is the schedule.
$q$ — the name of the forward-process distribution. $q (x_{t} ∣ x_{0})$ means "the probability spread of noisy point $x_{t}$ , given that we started from the specific clean point $x_{0}$ ." It is conditional on $x_{0}$ — we fix one image and ask where noise carries it.

Step 1 — One noising step, drawn as a shrink-then-shake

WHAT. A single forward step takes the current point $x_{t - 1}$ and produces the next one, using a fresh independent draw $ϵ_{t} \sim N (0, I)$ :

$next point x_{t} = shrink factor 1 - β_{t} current point x_{t - 1} + noise size β_{t} fresh bell-curve noise ϵ_{t}$

Read it left to right: shrink the old point a little (multiply by something just under $1$ ), then shake it by adding a scaled dab of noise.

WHY these two pieces and not one. If we only added noise ( $x_{t} = x_{t - 1} + β_{t} ϵ_{t}$ ), the cloud of possible points would keep swelling forever — the picture would fly off to infinity. The shrink factor $1 - β_{t}$ pulls the point back toward zero by just enough to cancel the swelling. That balance is the whole trick, and Step 3 proves it exactly.

PICTURE. The clean point (mint) is nudged toward the origin along the dashed grey line, then a lavender noise arrow scatters it to the new coral point.

Figure — Denoising diffusion probabilistic models (DDPM)

Step 2 — Why "variance" is the right ruler

WHAT. To track how big the cloud of possible $x_{t}$ gets, we measure its variance — the average squared distance from the centre.

WHY variance and not, say, the maximum. Because Gaussians (bell curves) are completely described by two things: their centre (mean) and their spread (variance). If we control the mean and the variance, we control the shape of the whole cloud. So a full derivation must pin down both — we do the mean in Step 3b and the variance in Step 3a.

PICTURE. Three bell curves of increasing spread — variance is just the width of the bell.

Two rules we will lean on:

Scaling: stretching a variable by $c$ multiplies its variance by $c^{2}$ , and multiplies its mean by $c$ . So $Var [c z] = c^{2} Var [z]$ and $E [c z] = c E [z]$ .
Adding independent noises: variances add (not the standard deviations). $Var [a + b] = Var [a] + Var [b]$ when $a, b$ are independent; means always add, $E [a + b] = E [a] + E [b]$ .

Step 3 — The shrink factor freezes the variance (and where the mean goes)

Setting the frame. Everything here is computed conditional on a fixed clean point $x_{0}$ — that is exactly what the symbol $q (\cdot ∣ x_{0})$ means. So "variance" below is the spread induced purely by the random noise draws, holding $x_{0}$ fixed. It is not the variance across different images.

WHAT (3a — variance). Suppose the current point has noise-induced spread $σ_{t - 1}^{2} = Var [x_{t} ∣ x_{0}]$ at the previous step. Apply the Step 2 rules to the Step 1 formula (remember $ϵ_{t}$ is independent of $x_{t - 1}$ ):

$Var [x_{t} ∣ x_{0}] = from (1 - β_{t})^{2} (1 - β_{t}) prev. spread σ_{t - 1}^{2} + from (β_{t})^{2} \cdot 1 β_{t} .$

Note the honest edge case: at the very first step, $x_{0}$ is a fixed number, so $σ_{0}^{2} = Var [x_{0} ∣ x_{0}] = 0$ , and the formula gives $Var [x_{1} ∣ x_{0}] = β_{1}$ — exactly the noise just injected, as it should be. The clean " $= 1$ " balance is the unconditional picture (averaging over data too), where our normalization makes $Var [x_{0}] = 1$ ; then:

$Var [x_{t}] = (1 - β_{t}) \cdot 1 + β_{t} = 1.$

So the shrink factor is tuned so the overall cloud (data spread + noise) stays unit-sized at every step; the conditional spread grows from $0$ up toward $1$ as the image is forgotten.

WHAT (3b — mean). Take the expectation of the Step 1 formula. Since $E [ϵ_{t}] = 0$ , the noise term vanishes in the mean:

$E [x_{t} ∣ x_{0}] = 1 - β_{t} E [x_{t - 1} ∣ x_{0}] + β_{t} = 0 E [ϵ_{t}] = α_{t} E [x_{t - 1} ∣ x_{0}] .$

Starting from $E [x_{0} ∣ x_{0}] = x_{0}$ and multiplying one shrink factor per step, the mean is $α_{1} α_{2} \dots α_{t} x_{0}$ . This is the $\overset{α}{ˉ}_{t} x_{0}$ in the closed form — now derived, not guessed.

WHY this matters. A Gaussian needs both a mean and a variance. We have now shown where each comes from: the mean is $x_{0}$ dragged toward zero by the accumulated shrink; the variance is the accumulated noise.

PICTURE. A see-saw: the shrink pan and the noise pan carry equal weight $β_{t}$ in the unconditional picture, so the bar stays level at variance $= 1$ .

Step 4 — Chain two steps and watch the noises merge

WHAT. Now push through two steps, each with its own independent draw ( $ϵ_{1}, ϵ_{2} \sim N (0, I)$ , independent):

\mathbf{x}_2 = \sqrt{\alpha_2}\,\mathbf{x}_1 + \sqrt{1-\alpha_2}\,\boldsymbol\epsilon_2.$$ Substitute $\mathbf{x}_1$ into $\mathbf{x}_2$ and group: $$\mathbf{x}_2 = \underbrace{\sqrt{\alpha_2\alpha_1}\,\mathbf{x}_0}_{\text{shrunk clean part}} \;+\; \underbrace{\sqrt{\alpha_2(1-\alpha_1)}\,\boldsymbol\epsilon_1 + \sqrt{1-\alpha_2}\,\boldsymbol\epsilon_2}_{\text{two independent noise arrows}}.$$ **WHY we can merge the two noise arrows.** Here is the key fact that makes DDPM tractable: > [!formula] Sum of independent Gaussians > Two independent bell-curve noises with spreads $a$ and $b$ add up to *one* bell-curve noise with spread $\sqrt{a^2+b^2}$: > $$a\,\boldsymbol\epsilon_1 + b\,\boldsymbol\epsilon_2 = \sqrt{a^2+b^2}\ \boldsymbol\epsilon.$$ So the two noise terms become a single $\sqrt{\alpha_2(1-\alpha_1)+(1-\alpha_2)}\,\boldsymbol\epsilon$. Simplify the inside: $$\alpha_2(1-\alpha_1)+(1-\alpha_2)=\alpha_2-\alpha_2\alpha_1+1-\alpha_2 = 1-\alpha_1\alpha_2.$$ Therefore $\ \mathbf{x}_2 = \sqrt{\alpha_1\alpha_2}\,\mathbf{x}_0 + \sqrt{1-\alpha_1\alpha_2}\,\boldsymbol\epsilon.$ **PICTURE.** Two lavender noise arrows placed tip-to-tail (they are perpendicular, because independent) combine by the Pythagorean rule into one longer coral arrow of length $\sqrt{a^2+b^2}$. ![[deepdives/dd-ai-ml-4.5.11-d2-s04.png]] --- ## Step 5 — Induction: the pattern holds for every $t$ **WHAT.** We now prove the general claim by **induction** — a domino argument: show it holds at the start, then show each true case forces the next one true. > [!definition] $\bar\alpha_t$ — the total surviving signal > $$\bar\alpha_t = \alpha_1\alpha_2\cdots\alpha_t = \prod_{s=1}^{t}\alpha_s.$$ > It is the fraction of the *original picture* still present at step $t$. It only ever shrinks toward $0$. **The induction hypothesis** (the claim for step $t-1$): $$\mathbf{x}_{t-1} = \sqrt{\bar\alpha_{t-1}}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha_{t-1}}\,\boldsymbol\epsilon',\qquad \boldsymbol\epsilon'\sim\mathcal{N}(0,\mathbf{I}).$$ **Base case ($t=1$).** Step 1 with $\bar\alpha_1=\alpha_1$ gives $\mathbf{x}_1=\sqrt{\alpha_1}\mathbf{x}_0+\sqrt{1-\alpha_1}\,\boldsymbol\epsilon_1$ — the claim holds. ✔ **Inductive step.** Feed the hypothesis into the Step 1 recurrence $\mathbf{x}_t=\sqrt{\alpha_t}\,\mathbf{x}_{t-1}+\sqrt{1-\alpha_t}\,\boldsymbol\epsilon_t$: $$\mathbf{x}_t = \underbrace{\sqrt{\alpha_t\bar\alpha_{t-1}}\,\mathbf{x}_0}_{=\,\sqrt{\bar\alpha_t}\,\mathbf{x}_0} \;+\; \underbrace{\sqrt{\alpha_t(1-\bar\alpha_{t-1})}\,\boldsymbol\epsilon' + \sqrt{1-\alpha_t}\,\boldsymbol\epsilon_t}_{\text{two independent noises} \to \text{merge}}.$$ Merge the two independent noises exactly as in Step 4 (Pythagoras): $$\alpha_t(1-\bar\alpha_{t-1})+(1-\alpha_t)=\alpha_t-\alpha_t\bar\alpha_{t-1}+1-\alpha_t = 1-\alpha_t\bar\alpha_{t-1}=1-\bar\alpha_t.$$ So $\ \mathbf{x}_t = \sqrt{\bar\alpha_t}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha_t}\,\boldsymbol\epsilon$. The claim at step $t$ holds — the domino falls. ∎ **WHY this closes the gap.** We did not just *assert* "the same merge happens every step" — we showed the merge at step $t$ produces exactly the shape needed to trigger step $t+1$. Combined with the base case, this proves it for all $t$. Because $\boldsymbol\epsilon$ is standard Gaussian, the mean is $\sqrt{\bar\alpha_t}\,\mathbf{x}_0$ (Step 3b) and the variance is $1-\bar\alpha_t$ (the merged noise size squared), so as a **distribution** this reads: $$q(\mathbf{x}_t\mid\mathbf{x}_0)=\mathcal{N}\big(\mathbf{x}_t;\ \underbrace{\sqrt{\bar\alpha_t}\,\mathbf{x}_0}_{\text{mean, from Step 3b}},\ \underbrace{(1-\bar\alpha_t)\mathbf{I}}_{\text{variance, from the merge}}\big).$$ **WHY it is the teleporter.** No sum over intermediate steps survives — only the single product $\bar\alpha_t$. To make a training example at any $t$, you compute $\bar\alpha_t$ once and draw one noise blob. **One line, no loop.** **PICTURE.** A "signal vs noise" mixing bar: at step $0$ it is all mint (signal), as $t$ grows the mint fraction is $\sqrt{\bar\alpha_t}$ and the coral (noise) fraction is $\sqrt{1-\bar\alpha_t}$. ![[deepdives/dd-ai-ml-4.5.11-d2-s05.png]] --- ## Step 6 — The two boundary indices: $t=0$ and $t=T$ **WHAT.** Check the formula at its extreme integer indices. > [!definition] The empty product $\bar\alpha_0$ > $\bar\alpha_0$ is a product with **no factors in it** (the product $\prod_{s=1}^{0}$ runs over nothing). By the standard convention an empty product equals $1$, so $\bar\alpha_0 = 1$. This is not a trick — it is what "start before any noise is added" must mean. - **At $t=0$:** with $\bar\alpha_0 = 1$ we get $\sqrt{\bar\alpha_0}=1$, $\sqrt{1-\bar\alpha_0}=0$, so $\mathbf{x}_0=\mathbf{x}_0$. Zero noise. ✔ The formula correctly does nothing. - **At $t=T$ (the last, large index):** each $\alpha_s<1$, so multiplying hundreds of them drives $\bar\alpha_T$ very close to $0$. Then $\sqrt{\bar\alpha_T}\approx 0$ and $\sqrt{1-\bar\alpha_T}\approx 1$, giving $\mathbf{x}_T\approx\boldsymbol\epsilon$ — pure standard noise, no memory of the image. **WHY these are exactly what we need.** The reverse (generation) process starts from $\mathbf{x}_T\sim\mathcal{N}(0,\mathbf{I})$. Step 6 guarantees the forward process actually *reaches* that pure-noise state, so sampling can begin from plain Gaussian noise with no bias. **PICTURE.** The curve $\bar\alpha_t$ sliding from $1$ down toward $0$ across timesteps, with the two boundary indices marked "all image" and "all noise". ![[deepdives/dd-ai-ml-4.5.11-d2-s06.png]] > [!mistake] The classic sign-slip > The clean-image factor is $\sqrt{\bar\alpha_t}$ and the noise factor is $\sqrt{1-\bar\alpha_t}$ — **not** the reverse. As noise takes over, the *image* factor shrinks. If your image gets *sharper* as $t$ grows, you swapped them. --- ## The one-picture summary ![[deepdives/dd-ai-ml-4.5.11-d2-s07.png]] One clean point, one noise blob, one knob $\bar\alpha_t$ that slides the mix from all-signal to all-noise — that is the entire forward process. > [!recall]- Feynman retelling — say it back in plain words > We wanted to dirty a picture with noise a thousand tiny times, but walking all thousand steps for every training example is wasteful. So we noticed each step does the same simple thing: shrink the current point a hair (multiply by a number just under one) and add a small dab of *fresh, independent* bell-curve noise. Working conditional on one fixed image $\mathbf{x}_0$, we found the mean of the noisy point is $\mathbf{x}_0$ dragged toward zero by the accumulated shrink, and the variance is the accumulated noise. We tuned the shrink so the overall cloud (once we also average over data, which we normalized to unit spread) never grows — it stays spread one, forever. Then we chained two steps and saw the two noise dabs merge into a single dab (independent Gaussians add like the sides of a right triangle — Pythagoras). We turned that one merge into a full domino proof by induction: the merged shape at step $t$ is exactly what triggers step $t+1$. The clean-image factors just multiplied together, so we bundled them into one running product $\bar\alpha_t$. End result: to reach step $t$, take $\sqrt{\bar\alpha_t}$ of the original picture plus $\sqrt{1-\bar\alpha_t}$ of one fresh noise blob — this is $q(\mathbf{x}_t\mid\mathbf{x}_0)$, the forward distribution. Check the boundary indices: at $t=0$ the empty product makes $\bar\alpha_0=1$ so nothing changes, at index $T$ the picture is gone and we have pure noise — exactly the noise we sample from when generating. Teleporter proven honest. > [!recall]- > What does $\bar\alpha_t$ mean physically? ::: The fraction of the original image still surviving at step $t$; it slides from $1$ down to nearly $0$. > What does the symbol $q(\mathbf{x}_t\mid\mathbf{x}_0)$ mean? ::: The forward-process distribution — the spread of the noisy point $\mathbf{x}_t$, computed conditional on a fixed clean start $\mathbf{x}_0$. > Where does the mean $\sqrt{\bar\alpha_t}\,\mathbf{x}_0$ come from? ::: Taking the expectation step by step: noise has zero mean, so only the shrink factors survive and multiply into $\sqrt{\bar\alpha_t}$. > Why is there a $\sqrt{1-\beta_t}$ shrink factor at all? ::: To cancel the growth in variance from added noise, keeping the overall (unconditional) spread fixed at $1$ every step. > How do two independent noise arrows of sizes $a$ and $b$ combine? ::: Into one of size $\sqrt{a^2+b^2}$ — the Pythagorean rule. > Why is $\bar\alpha_0=1$? ::: It is an empty product (no factors), which by convention equals $1$ — the "before any noise" state. > What is $\mathbf{x}_T$ at the last index? ::: Essentially pure standard Gaussian noise $\mathcal{N}(0,\mathbf{I})$, since $\bar\alpha_T\to 0$. **See also:** [[Variational Autoencoders (VAE)]] for the reparameterization trick used here, [[Markov Chain Monte Carlo]] for the Markov-chain view, and the [[U-Net architecture]] that implements $\boldsymbol\epsilon_\theta$.