Foundations — Diffusion models forward - reverse process

This page is the toolbox. Before you touch the parent note's equations, you need to own every symbol in them. We build each one from a picture, in an order where nothing appears before it is earned.

1. What is a "sample" and a "distribution"? ($x_0$, $q$)

Everything starts with data. Call one real image $x_0$ — the subscript $0$ just means "step zero, before we ruin anything".

Now imagine plotting every real image as a dot. Real images clump together (faces near faces, digits near digits). That cloud of dots is the data distribution, written $q(x_0)$.

Why the topic needs it: generation means producing a fresh dot that lands inside the thick fog — a new image that looks real. Diffusion's whole trick is a clever route to that fog. (This is exactly the goal that Variational autoencoders also chase, by a different road.)

2. The Gaussian — the shape of "random noise" ($\mathcal{N}$, mean, variance, $I$)

The single most-used symbol in the parent note is $\mathcal{N}$.

Two words to lock in, because the parent uses them constantly:

• Mean $\mu$ (Greek "mu") = the centre / average / balance point of the hill.
• Variance ${\sigma }^2$ (Greek "sigma" squared) = the spread. Its square root $\sigma$ is the typical distance from centre. Small variance = tight spike; large variance = flat spread.

Why "$\sqrt{\text{variance}}$" keeps appearing: if a number has variance ${\sigma }^2$, its typical size is $\sigma =\sqrt{{\sigma }^2}$. So $\sqrt{{\beta }_t}$ in the parent is literally "the typical strength of the noise added at step $t$".

What is that $I$?

An image is many numbers, not one. So the noise is a bell curve in many dimensions at once. We write $\mathcal{N}(\mu ,\Sigma )$ where $\Sigma$ (capital sigma) is now a table saying how each dimension varies and co-varies.

Why the topic needs it: diffusion adds independent noise to every pixel with the same strength, which is exactly what ${\beta }_{t}I$ encodes — cheap to write, cheap to sample.

3. The reparameterization idea ($ϵ$, "$\sim$", $=$ vs sampling)

The parent writes both $x_t\sim \mathcal{N}(\cdots )$ and $x_t=\sqrt{\dots }{x}_0+\sqrt{\dots }ϵ$. These say the same thing two ways.

Here's the picture-based trick the whole topic leans on:

Why we bother: this turns "sample from a distribution" (mysterious) into "do arithmetic on one fixed noise draw $ϵ$" (a network can differentiate through it). $ϵ$ (Greek "epsilon") is the exact quantity the reverse network is asked to predict, so it deserves a picture. This same trick powers Variational autoencoders.

One fact we will lean on: adding two independent Gaussians

4. A chain of steps: $t$, $T$, and the Markov property

The subscript $t$ counts noise steps: $x_0$ (clean) $\to x_1\to x_2\to \cdots \to x_T$ (pure static). $T$ is the total, typically $1000$.

Why is it called a Markov chain? Because each new frame depends only on the frame just before it, not the whole history.

Why the topic needs it: memorylessness is what lets us write the forward process as one simple rule $q(x_t\mid x_{t-1})$ repeated, and lets the reverse be a matching chain $p_{\theta }(x_{t-1}\mid x_t)$. If you want this idea rigorously, see Markov chains.

Conditional probability bar "$\mid$"

$q(x_t\mid x_{t-1})$ reads "the distribution of $x_t$ given that $x_{t-1}$ is already fixed". The bar $\mid$ means given / assuming we already know. Picture freezing the previous frame, then asking where the next frame can land.

5. The schedule: ${\beta }_t$, ${\alpha }_t$, ${\bar{\alpha }}_t$

These three control how fast the image dissolves. Get them straight and half the parent note's algebra evaporates.

Where the closed-form shortcut comes from

The parent quotes $x_t=\sqrt{{\bar{\alpha }}_t}{x}_0+\sqrt{1-{\bar{\alpha }}_t}ϵ$ as if by magic. Let us earn it, using only the one-step rule and the "Gaussians add" fact from §3.

One step, written in reparameterized form (recall ${\alpha }_t=1-{\beta }_t$): $x_t=\sqrt{{\alpha }_t}{x}_{t-1}+\sqrt{1-{\alpha }_t}{ϵ}_{t-1},{ϵ}_{t-1}\sim \mathcal{N}(0,I).$ WHAT this says: keep a fraction $\sqrt{{\alpha }_t}$ of the previous frame, add a fresh independent shake of spread $\sqrt{1-{\alpha }_t}$.

Now substitute the same rule for $x_{t-1}$ (this is legal because the chain is memoryless): $x_t=\sqrt{{\alpha }_t}(\sqrt{{\alpha }_{t-1}}{x}_{t-2}+\sqrt{1-{\alpha }_{t-1}}{ϵ}_{t-2})+\sqrt{1-{\alpha }_t}{ϵ}_{t-1}.$ Multiply the signal term through: $x_t=\sqrt{{\alpha }_t{\alpha }_{t-1}}{x}_{t-2}+\underset{\text{two independent Gaussians}}{\underbrace{\sqrt{{\alpha }_t(1-{\alpha }_{t-1})}{ϵ}_{t-2}+\sqrt{1-{\alpha }_t}{ϵ}_{t-1}}}.$

WHY we now use the "Gaussians add" fact: the two noise terms are independent, so by §3 they merge into a single Gaussian. Their variances are ${\alpha }_t(1-{\alpha }_{t-1})$ and $(1-{\alpha }_t)$; adding them: ${\alpha }_t(1-{\alpha }_{t-1})+(1-{\alpha }_t)={\alpha }_t-{\alpha }_t{\alpha }_{t-1}+1-{\alpha }_t=1-{\alpha }_t{\alpha }_{t-1}.$ So the merged noise has spread $\sqrt{1-{\alpha }_t{\alpha }_{t-1}}$ and: $x_t=\sqrt{{\alpha }_t{\alpha }_{t-1}}{x}_{t-2}+\sqrt{1-{\alpha }_t{\alpha }_{t-1}}\overline{ϵ}.$

WHAT IT LOOKS LIKE: exactly the same shape as one step, but with the signal factor now the product ${\alpha }_t{\alpha }_{t-1}$. Repeat this collapse all the way down to $x_0$: every extra step multiplies one more ${\alpha }_s$ onto the signal factor, and the noise variance stays "$1$ minus that product". After $t$ collapses the product is precisely ${\bar{\alpha }}_t={\prod }_{s=1}^t{\alpha }_s$.

Why the topic needs it: ${\bar{\alpha }}_t$ is the shortcut that lets you jump straight to any noise level from $x_0$ with no loop — the engine of efficient training. Different choices of the ${\beta }_t$ dial also underpin faster samplers like DDIM sampling.

6. The reverse chain: $p_{\theta }$, $\theta$, ${ϵ}_{\theta }$, ${\mu }_{\theta }$

The forward chain $q$ destroys; the reverse chain rebuilds. It is written $p_{\theta }(x_{t-1}\mid x_t)$.

How does the network compute that centre? It does not guess ${\mu }_{\theta }$ directly — it guesses the noise $ϵ$ and plugs it into a formula.

Why the topic needs it: the reverse chain $p_{\theta }$ is the generator. Sampling from it repeatedly, $x_T\to x_{T-1}\to \cdots \to x_0$, is how a fresh image is born. Steering that guess is what Classifier-free guidance does.

7. The training goal: expectation $\mathbb{E}$ and norm $\Vert \cdot {\Vert }^2$

Why the topic needs it: this single mean-squared-error is the entire objective. Everything else — the reverse mean ${\mu }_{\theta }$, the sampling loop — is bookkeeping around "predict the noise". The link to ${\nabla }_x\log q(x)$ (a score) connects this to Langevin dynamics.

Prerequisite map

Equipment checklist

Cover the right side and answer aloud; reveal to check.