Exercises — Denoising diffusion probabilistic models (DDPM)

(a) Mean $=\sqrt{1-{\beta }_t}{\mathbf{x}}_{t-1}$. The $\mathcal{N}(\cdot ;\text{mean},\text{var})$ notation always lists the mean in the middle slot. (b) Variance $={\beta }_t\mathbf{I}$ — the identity matrix $\mathbf{I}$ means noise is added independently and equally to every pixel. (c) Because $0<{\beta }_t<1$ makes $\sqrt{1-{\beta }_t}<1$. Each step shrinks the old signal toward zero and replaces the lost "energy" with fresh noise, so total variance stays near $1$ instead of exploding.

${\alpha }_t=1-{\beta }_t$ so ${\alpha }_1=0.9,{\alpha }_2=0.8,{\alpha }_3=0.7$. ${\bar{\alpha }}_3={\alpha }_1{\alpha }_2{\alpha }_3=0.9\times 0.8\times 0.7=0.504$. Reading: after 3 steps only $50.4\%$ of the original signal scale survives. (Verified.)

$\sqrt{{\bar{\alpha }}_t}=\sqrt{0.36}=0.6$, and $\sqrt{1-{\bar{\alpha }}_t}=\sqrt{0.64}=0.8$. ${\mathbf{x}}_t=0.6\times 0.6+0.8\times 1.5=0.36+1.2=1.56$. The signal contributes $0.36$; the noise dominates with $1.2$. (Verified.)

${\mathbf{x}}_0=\frac{1.56-0.8\times 1.5}{0.6}=\frac{1.56-1.2}{0.6}=\frac{0.36}{0.6}=\mathrm{0.6.}$ We get back the original $0.6$ exactly. This is the whole idea of noise-prediction: if the network can name the $ϵ$ that was added, algebra recovers the clean pixel. (Verified.)

Linear means ${\beta }_t={\beta }_1+\frac{t-1}{T-1}({\beta }_T-{\beta }_1)$. At $t=500$: fraction $=\frac{499}{999}\approx 0.4995$. ${\beta }_{500}\approx 0.0001+0.4995\times (0.02-0.0001)=0.0001+0.4995\times 0.0199\approx 0.010042$. So the midpoint injects about $1\%$ variance per step. (Verified.)

${\mathbf{x}}_{t-1}$ and $ϵ$ are independent, so variances add after scaling: $\text{Var}[{\mathbf{x}}_t]=(\sqrt{{\alpha }_t}{)}^2\text{Var}[{\mathbf{x}}_{t-1}]+(\sqrt{{\beta }_t}{)}^2\text{Var}[ϵ].$ Substituting $\text{Var}[{\mathbf{x}}_{t-1}]=1$, $\text{Var}[ϵ]=1$: $={\alpha }_t\cdot 1+{\beta }_t\cdot 1={\alpha }_t+{\beta }_t=(1-{\beta }_t)+{\beta }_t=1.$ The variance is a fixed point at $1$. Why the design needs this: without it, variance would drift and the final ${\mathbf{x}}_T$ would not match the $\mathcal{N}(0,\mathbf{I})$ we sample from at generation time. (Verified.)

${\tilde{\beta }}_t=\frac{1-0.6}{1-0.48}\times 0.2=\frac{0.4}{0.52}\times 0.2\approx 0.7692\times 0.2\approx \mathrm{0.15385.}$ It is smaller than ${\beta }_t=0.2$ because the ratio $\frac{0.4}{0.52}<1$. Meaning: when we run the reverse step knowing ${\mathbf{x}}_0$, we are more certain than the raw forward noise level — conditioning on the endpoint shrinks the uncertainty. (Verified.)

Coefficient of ${\mathbf{x}}_0$: $\frac{\sqrt{0.6}\times 0.2}{0.52}=\frac{0.774597\times 0.2}{0.52}=\frac{0.154919}{0.52}\approx \mathrm{0.297922.}$ Coefficient of ${\mathbf{x}}_t$: $\frac{\sqrt{0.8}\times (1-0.6)}{0.52}=\frac{0.894427\times 0.4}{0.52}=\frac{0.357771}{0.52}\approx \mathrm{0.688021.}$ ${\tilde{\mathbf{\mu }}}_t\approx 0.297922\times 1.0+0.688021\times 0.5\approx 0.297922+0.344010\approx \mathrm{0.641932.}$ The posterior mean is a weighted blend of "where we came from" (${\mathbf{x}}_0$) and "where we are" (${\mathbf{x}}_t$). The figure below plots exactly this blend: the red dot (the mean) sits between the yellow ${\mathbf{x}}_0$ and the blue ${\mathbf{x}}_t$, pulled harder toward ${\mathbf{x}}_t$ because its weight ($\approx 0.688$) is the larger. (Verified.)

What we do & why: we want the mean to depend only on the known ${\mathbf{x}}_t$ and the predictable $ϵ$, not on the unknown clean ${\mathbf{x}}_0$. So we replace ${\mathbf{x}}_0$ using the inversion of the closed-form jump.

Step 1 — substitute. Put ${\mathbf{x}}_0=\frac{{\mathbf{x}}_t-\sqrt{1-{\bar{\alpha }}_t}ϵ}{\sqrt{{\bar{\alpha }}_t}}$ into the first term of ${\tilde{\mathbf{\mu }}}_t$: $\frac{\sqrt{{\bar{\alpha }}_{t-1}}{\beta }_t}{1-{\bar{\alpha }}_t}\cdot \frac{{\mathbf{x}}_t-\sqrt{1-{\bar{\alpha }}_t}ϵ}{\sqrt{{\bar{\alpha }}_t}}.$ Step 2 — use the product identity ${\bar{\alpha }}_t={\alpha }_t{\bar{\alpha }}_{t-1}$, hence $\frac{\sqrt{{\bar{\alpha }}_{t-1}}}{\sqrt{{\bar{\alpha }}_t}}=\frac{1}{\sqrt{{\alpha }_t}}$. The first term becomes $\frac{{\beta }_t}{(1-{\bar{\alpha }}_t)\sqrt{{\alpha }_t}}({\mathbf{x}}_t-\sqrt{1-{\bar{\alpha }}_t}ϵ).$ Step 3 — collect the ${\mathbf{x}}_t$ coefficients. Add the ${\mathbf{x}}_t$ piece above to the second term of ${\tilde{\mathbf{\mu }}}_t$, whose coefficient is $\frac{\sqrt{{\alpha }_t}(1-{\bar{\alpha }}_{t-1})}{1-{\bar{\alpha }}_t}$. Total ${\mathbf{x}}_t$ coefficient: $\frac{{\beta }_t}{(1-{\bar{\alpha }}_t)\sqrt{{\alpha }_t}}+\frac{\sqrt{{\alpha }_t}(1-{\bar{\alpha }}_{t-1})}{1-{\bar{\alpha }}_t}=\frac{{\beta }_t+{\alpha }_t(1-{\bar{\alpha }}_{t-1})}{(1-{\bar{\alpha }}_t)\sqrt{{\alpha }_t}}.$ The numerator is ${\beta }_t+{\alpha }_t-{\alpha }_t{\bar{\alpha }}_{t-1}=(1-{\alpha }_t)+{\alpha }_t-{\bar{\alpha }}_t=1-{\bar{\alpha }}_t$ (using ${\beta }_t=1-{\alpha }_t$ and ${\alpha }_t{\bar{\alpha }}_{t-1}={\bar{\alpha }}_t$). So the ${\mathbf{x}}_t$ coefficient collapses to $\frac{1-{\bar{\alpha }}_t}{(1-{\bar{\alpha }}_t)\sqrt{{\alpha }_t}}=\frac{1}{\sqrt{{\alpha }_t}}.$ Step 4 — collect the $ϵ$ coefficient. Only the first term carries $ϵ$: $-\frac{{\beta }_t}{(1-{\bar{\alpha }}_t)\sqrt{{\alpha }_t}}\cdot \sqrt{1-{\bar{\alpha }}_t}=-\frac{1}{\sqrt{{\alpha }_t}}\cdot \frac{{\beta }_t}{\sqrt{1-{\bar{\alpha }}_t}}.$ Step 5 — assemble. Combining Steps 3 and 4: $\mathbf{\mu }=\frac{1}{\sqrt{{\alpha }_t}}({\mathbf{x}}_t-\frac{{\beta }_t}{\sqrt{1-{\bar{\alpha }}_t}}ϵ).$

Numeric sanity check. Take ${\alpha }_t=0.8,{\bar{\alpha }}_t=0.48,{\bar{\alpha }}_{t-1}=0.6,{\beta }_t=0.2,{\mathbf{x}}_t=0.5,ϵ=0.5$.

• New form: $\frac{1}{\sqrt{0.8}}(0.5-\frac{0.2}{\sqrt{0.52}}\times 0.5)=1.118034\times (0.5-0.138675)\approx \mathrm{0.403975.}$
• Original via ${\mathbf{x}}_0$: ${\mathbf{x}}_0=\frac{0.5-\sqrt{0.52}\times 0.5}{\sqrt{0.48}}\approx 0.201271$; then $0.297922\times 0.201271+0.688021\times 0.5\approx \mathrm{0.403974.}$ Both equal $\approx 0.40397$ — the reparameterisation is exact. (Verified.)

Why bother? Predicting $ϵ$ is a fixed-scale, unit-variance target at every $t$, which trains far more stably than predicting ${\mathbf{x}}_0$ or $\mathbf{\mu }$ whose scale drifts with $t$. This is the bridge to $L_{\text{simple}}=\mathbb{E}\Vert ϵ-{ϵ}_{\theta }{\Vert }^2$.

$\sqrt{0.542}\approx 0.736206,\sqrt{0.458}\approx \mathrm{0.676757.}$ ${\mathbf{x}}_{437}=0.736206\times 0.6+0.676757\times (-1.0)=0.441724-0.676757\approx -\mathrm{0.235033.}$ Loss $=(ϵ-{ϵ}_{\theta }{)}^2=(-1.0-(-0.7){)}^2=(-0.3{)}^2=\mathrm{0.09.}$ That $0.09$ is what backpropagation drives toward $0$. (Verified.)

(a) ${\bar{\alpha }}_1=0.9$, ${\bar{\alpha }}_2=0.9\times 0.8=0.72$, ${\bar{\alpha }}_3=0.72\times 0.7=0.504$. (b) $\sqrt{0.504}\approx 0.709930$ (signal), $\sqrt{1-0.504}=\sqrt{0.496}\approx 0.704273$ (noise). (c) At $T=3$ the signal still contributes $\approx 0.71{\mathbf{x}}_0$ — the image is very much not pure noise. For ${\mathbf{x}}_T$ to be indistinguishable from $\mathcal{N}(0,\mathbf{I})$ we need ${\bar{\alpha }}_T\to 0$, which is why real DDPMs use $T\approx 1000$ with tiny ${\beta }_t$. (Verified.)

(i) VAE — the training loss is a variational lower bound on $-\log p_{\theta }({\mathbf{x}}_0)$, exactly the VAE idea of bounding an intractable likelihood. (ii) MCMC — the reverse process is a Markov chain whose repeated sampling walks noise back to data, echoing sampling-by-chain. (iii) U-Net — the network ${ϵ}_{\theta }({\mathbf{x}}_t,t)$ is almost always a U-Net, chosen because denoising needs both fine detail (skip connections) and global context (bottleneck). (iv) Classifier-free guidance — steers ${ϵ}_{\theta }$ toward a text/class condition by mixing conditional and unconditional predictions. (v) Latent diffusion — runs the whole DDPM machinery in a compressed latent space instead of pixels, for speed.

The mean is $\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0$. As ${\bar{\alpha }}_t\to 0$, $\sqrt{{\bar{\alpha }}_t}\to 0$, so mean $\to 0$ for any finite ${\mathbf{x}}_0$. The variance is $(1-{\bar{\alpha }}_t)\mathbf{I}$. As ${\bar{\alpha }}_t\to 0$, this $\to \mathbf{I}$. Therefore $q({\mathbf{x}}_t\mid {\mathbf{x}}_0)\to \mathcal{N}(0,\mathbf{I})$ — independent of the starting image. This is the guarantee that lets generation begin from a fresh $\mathcal{N}(0,\mathbf{I})$ draw with no memory of any particular datum. (Verified.)

Is ${\bar{\alpha }}_t$ a sum or a product of the ${\alpha }_s$? ::: A product, ${\bar{\alpha }}_t={\prod }_{s=1}^t{\alpha }_s$. Why scale ${\mathbf{x}}_{t-1}$ by $\sqrt{1-{\beta }_t}$? ::: To keep total variance $\approx 1$ (variance-preserving). What does the network ${ϵ}_{\theta }$ predict? ::: The noise $ϵ$ that was added, not the clean image. As ${\bar{\alpha }}_t\to 0$, $q({\mathbf{x}}_t\mid {\mathbf{x}}_0)\to$ what? ::: $\mathcal{N}(0,\mathbf{I})$, standard Gaussian.