Question bank — Denoising diffusion probabilistic models (DDPM)

${\mathbf{x}}_t=\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0+\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ}$ holds — walk it, don't memorize it What we want: skip all the intermediate steps and land at ${\mathbf{x}}_t$ straight from ${\mathbf{x}}_0$. Step 1 (one step): ${\mathbf{x}}_1=\sqrt{{\alpha }_1}{\mathbf{x}}_0+\sqrt{1-{\alpha }_1}{\mathbf{ϵ}}_1$. Step 2 (compose): ${\mathbf{x}}_2=\sqrt{{\alpha }_2}{\mathbf{x}}_1+\sqrt{1-{\alpha }_2}{\mathbf{ϵ}}_2$. Substitute ${\mathbf{x}}_1$: ${\mathbf{x}}_2=\sqrt{{\alpha }_2{\alpha }_1}{\mathbf{x}}_0+\underset{\text{two independent Gaussians}}{\underbrace{\sqrt{{\alpha }_2(1-{\alpha }_1)}{\mathbf{ϵ}}_1+\sqrt{1-{\alpha }_2}{\mathbf{ϵ}}_2}}.$ Step 3 (the key merge): two independent zero-mean Gaussians add their variances, so the noisy tail collapses to a single Gaussian with variance ${\alpha }_2(1-{\alpha }_1)+(1-{\alpha }_2)=1-{\alpha }_1{\alpha }_2$. Hence ${\mathbf{x}}_2=\sqrt{{\alpha }_1{\alpha }_2}{\mathbf{x}}_0+\sqrt{1-{\alpha }_1{\alpha }_2}\mathbf{ϵ}$. Step 4 (induction): the same "signal shrinks by $\sqrt{\alpha }$, noise fills the rest to keep variance $1$" pattern repeats, giving ${\bar{\alpha }}_t={\prod }_{s=1}^t{\alpha }_s$ in front of ${\mathbf{x}}_0$ and $\sqrt{1-{\bar{\alpha }}_t}$ in front of the merged noise. Intuition: the signal-share ${\bar{\alpha }}_t$ and the noise-share $1-{\bar{\alpha }}_t$ always sum to $1$ — the picture below shows one bar splitting into signal (red) and noise (black) as $t$ grows.

${\tilde{\mathbf{\mu }}}_t$ and ${\tilde{\beta }}_t$ — where they come from What: conditioned on the clean data ${\mathbf{x}}_0$, the reverse step $q({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t,{\mathbf{x}}_0)=\mathcal{N}({\mathbf{x}}_{t-1};{\tilde{\mathbf{\mu }}}_t,{\tilde{\beta }}_t\mathbf{I})$ is Gaussian. Why Gaussian: by Bayes, $q({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t,{\mathbf{x}}_0)\propto q({\mathbf{x}}_t\mid {\mathbf{x}}_{t-1})q({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_0)$ — a product of two Gaussians in ${\mathbf{x}}_{t-1}$, and a product of Gaussians is again Gaussian (add the two "$-\frac{1}{2}(\cdot {)}^2$" exponents and complete the square). The parameters that fall out of completing the square are ${\tilde{\beta }}_t=\frac{1-{\bar{\alpha }}_{t-1}}{1-{\bar{\alpha }}_t}{\beta }_t,{\tilde{\mathbf{\mu }}}_t=\frac{\sqrt{{\bar{\alpha }}_{t-1}}{\beta }_t}{1-{\bar{\alpha }}_t}{\mathbf{x}}_0+\frac{\sqrt{{\alpha }_t}(1-{\bar{\alpha }}_{t-1})}{1-{\bar{\alpha }}_t}{\mathbf{x}}_t.$ Read it: ${\tilde{\mathbf{\mu }}}_t$ is a weighted blend of "where we came from" (${\mathbf{x}}_0$) and "where we are" (${\mathbf{x}}_t$); the network's job is to reproduce this blend without being told ${\mathbf{x}}_0$.

$\mathbf{ϵ}$ and predicting ${\mathbf{x}}_0$ are the same parameterization (invertible linear map) Start from the forward closed form: ${\mathbf{x}}_t=\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0+\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ}$. Solve for ${\mathbf{x}}_0$ (pure algebra — move terms, divide): ${\mathbf{x}}_0=\frac{{\mathbf{x}}_t-\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ}}{\sqrt{{\bar{\alpha }}_t}}.$ Solve for $\mathbf{ϵ}$ the same way: $\mathbf{ϵ}=\frac{{\mathbf{x}}_t-\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0}{\sqrt{1-{\bar{\alpha }}_t}}.$ The point: given the known ${\mathbf{x}}_t$ and the fixed constants $\sqrt{{\bar{\alpha }}_t},\sqrt{1-{\bar{\alpha }}_t}$, each of ${\mathbf{x}}_0$ and $\mathbf{ϵ}$ is an affine (straight-line) function of the other — knowing one hands you the other exactly. So a network predicting $\mathbf{ϵ}$ implicitly predicts ${\mathbf{x}}_0$; they carry identical information. Empirically the $\mathbf{ϵ}$ target is unit-scale for every $t$, so it trains more stably.

Which of $q$ and $p_{\theta }$ is learned? ::: Only $p_{\theta }$ (reverse); $q$ (forward) is fixed. Is $q({\mathbf{x}}_{t-1}\mid {\mathbf{x}}_t)$ Gaussian? ::: Only when also conditioned on ${\mathbf{x}}_0$. Does ${\bar{\alpha }}_t$ grow or shrink with $t$? ::: Shrinks toward $0$. What is the transition $q({\mathbf{x}}_t\mid {\mathbf{x}}_{t-1})$? ::: $\mathcal{N}({\mathbf{x}}_t;\sqrt{{\alpha }_t}{\mathbf{x}}_{t-1},{\beta }_t\mathbf{I})$.