Foundations — Denoising diffusion probabilistic models (DDPM)

This page assumes you have seen the parent DDPM note and want every symbol in it explained from the ground up. We build the alphabet before we read the sentence. By the end, none of the parent's notation should look like hieroglyphics.

1. What is "data" here — the symbol $\mathbf{x}$

Picture a single pixel as a dot whose brightness is a number between $-1$ (black) and $+1$ (white). A whole image is just many such numbers side by side. Writing them bold, $\mathbf{x}$, lets us talk about the entire image with one letter instead of $784$ letters.

Figure s01 (below): what "image = vector" means. The left panel shows a tiny $4\times 4$ image; each cell holds one brightness number (printed on it). The magenta arrow shows us flattening it row by row into the single long list $\mathbf{x}=[0.9,0.2,0.6,0.1,0.3,\dots ]$ on the right. That list is the object every DDPM formula operates on.

The subscript on $\mathbf{x}$ is a clock reading, not a pixel index:

• ${\mathbf{x}}_0$ = the original clean image (time zero), normalized to unit variance.
• ${\mathbf{x}}_t$ = the same image after $t$ steps of noise added.
• ${\mathbf{x}}_T$ = fully noised, $T$ being the last step (e.g. $T=1000$).

So ${\mathbf{x}}_0\to {\mathbf{x}}_1\to \cdots \to {\mathbf{x}}_T$ is one image getting progressively destroyed.

2. Randomness — the symbol $\mathcal{N}$ and the bell curve

The "static" we add is not any old randomness; it is Gaussian (bell-curve) noise. We must earn that symbol.

Figure s02 (below): the shape of $\mathcal{N}$. Two bell curves are drawn. The magenta one is narrow ($\sigma =1$); the orange one is wide ($\sigma =2$). The violet dashed line marks the shared centre $\mu =0$. Notice both peak at the centre and fade symmetrically outward — "drawing a random number" means throwing a dart under a hill, landing near the peak most of the time. The width of the hill is controlled entirely by $\sigma$.

Why Gaussian and not, say, uniform noise? Two reasons the topic needs it:

1. Adding two Gaussians gives another Gaussian (they combine cleanly) — this is the property that lets the parent note jump from ${\mathbf{x}}_0$ to any ${\mathbf{x}}_t$ in one shot.
2. A neural net can describe a Gaussian with just two outputs (centre + spread), so each tiny denoising step is easy to model.

The bold version: $\mathcal{N}(\mathbf{x};\mathbf{\mu },\Sigma )$

When $\mathbf{x}$ is a whole image (many numbers), the centre $\mathbf{\mu }$ is also a whole image (bold), and the spread becomes a matrix $\Sigma$. In DDPM the spread is always $\beta \mathbf{I}$:

The semicolon in $\mathcal{N}(\mathbf{x};\mathbf{\mu },\Sigma )$ just reads: "the bell curve, evaluated at $\mathbf{x}$, that is centred at $\mathbf{\mu }$ with spread $\Sigma$."

3. The dials — ${\beta }_t$, ${\alpha }_t$, ${\bar{\alpha }}_t$

We now earn the three scheduling symbols before we use them in the reparameterized noising formula.

Figure s03 (below): the variance budget over time. The magenta curve is the signal fraction ${\bar{\alpha }}_t$; the orange curve is the noise fraction $1-{\bar{\alpha }}_t$; together they always fill the box up to the violet dashed line at height $1$. Read left to right: at $t=0$ the image is pure signal; by $t=T$ it is pure noise; and at every intermediate step the two shaded areas stack to exactly $1$ — the total spread never changes. That constant-$1$ line is the whole point of normalizing ${\mathbf{x}}_0$ (Section 1).

$\prod$ (multiply all) versus $\sum$ (capital sigma, "add them all") — the parent uses both; $\sum$ appears in the training loss where we add up errors over steps (Section 6).

4. The noise-adder — $\mathbf{ϵ}$ and the reparameterization trick

Now that ${\bar{\alpha }}_t$ exists, the parent's headline formula is legible.

The magic move the parent uses everywhere:

${\mathbf{x}}_t=\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0+\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ}$

Read left to right: shrink the clean image by $\sqrt{{\bar{\alpha }}_t}$, then add static scaled by $\sqrt{1-{\bar{\alpha }}_t}$.

Cover the limits:

• At $t=0$: ${\bar{\alpha }}_0=1$, so ${\mathbf{x}}_t={\mathbf{x}}_0$ — untouched image. ✓
• As $t\to T$: ${\bar{\alpha }}_t\to 0$, so ${\mathbf{x}}_t\to \mathbf{ϵ}$ — pure static. ✓
• In between: a smooth blend, signal fading, static rising, total spread constant at $1$.

5. Probability grammar — $q$, $p_{\theta }$, $\sim$, and the bar $|$

6. The measuring sticks — $\mathbb{E}$, $\Vert \cdot {\Vert }^2$, $D_{KL}$

7. Where the pieces live in the bigger picture

The network ${\mathbf{ϵ}}_{\theta }$ that eats a noisy image and predicts its static is a U-Net. The whole "corrupt-then-learn-to-reverse" philosophy is the same latent-variable idea as Variational Autoencoders (VAE), and the variational lower bound is inherited from it. Steering generation toward a prompt is Classifier-free guidance; running the whole diffusion inside a compressed space is Latent diffusion models. Prefer the Hindi-English walkthrough? See 4.5.11 Denoising diffusion probabilistic models (DDPM) (Hinglish).

The map below reads top to bottom as a dependency chart: each arrow means "you need this before that." Follow it as a study order.

• The three inputs on the left — the vector $\mathbf{x}$ (Section 1), the Gaussian $\mathcal{N}$ (Section 2), and the schedule $\beta ,\alpha ,\bar{\alpha }$ (Section 3) — all feed the forward process $q$.
• The Markov property (one-step memory) also feeds $q$ and is what makes the reverse posterior $q({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t,{\mathbf{x}}_0)$ usable.
• The forward process gives us the reparameterized jump to any ${\mathbf{x}}_t$, which feeds the reverse posterior, which — together with the expectation/norm and KL measuring sticks (Section 6, KL borrowed from the VAE bound) — collapses into the simple "predict $\mathbf{ϵ}$" loss.
• That loss, plus the U-Net playing the role of ${\mathbf{ϵ}}_{\theta }$, is what DDPM training and sampling run on.

Equipment checklist

Test yourself — reveal only after you have answered aloud.