Exercises — Diffusion models forward - reverse process

• ${\beta }_t$ ::: the variance of the fresh noise added at step $t$ (small, e.g. $10^{-4}$ to $0.02$).
• ${\alpha }_t=1-{\beta }_t$ ::: the fraction of the previous signal's variance kept after step $t$.
• ${\bar{\alpha }}_t={\prod }_{s=1}^t{\alpha }_s$ ::: the total fraction of the original signal surviving after $t$ steps — a running product of all keep-fractions.

WHAT: convert each $\beta$ to a keep-fraction, then multiply them. ${\alpha }_1=1-0.1=0.9,{\alpha }_2=0.8,{\alpha }_3=0.7$ WHY multiply: signal surviving three steps is (survive step 1) $\times$ (survive step 2) $\times$ (survive step 3). ${\bar{\alpha }}_3=0.9\times 0.8\times 0.7=0.504$ So about $50.4\%$ of the original signal variance remains after three steps.

• The network ${ϵ}_{\theta }$ predicts ==the noise $ϵ$== that was mixed into $x_t$ — not the clean image.
• $\Vert \cdot {\Vert }^2$ is the squared Euclidean distance: sum over all pixels of (true noise $-$ predicted noise)${}^2$. It is $0$ only when every pixel's noise guess is exact.

WHAT: plug into $x_t=\sqrt{{\bar{\alpha }}_t}{x}_0+\sqrt{1-{\bar{\alpha }}_t}ϵ$. $\sqrt{{\bar{\alpha }}_t}=\sqrt{0.36}=0.6,\sqrt{1-{\bar{\alpha }}_t}=\sqrt{0.64}=0.8$ $x_t=0.6x_0+0.8ϵ$ Read it off the figure: mint bar $=0.6$, coral bar $=0.8$, and $0.6^2+0.8^2=1$. ✔

WHY this ratio: it directly compares the two bars in our figure — how loud the signal is versus the noise. $\text{SNR}(0.87)=\sqrt{\frac{0.87}{0.13}}=\sqrt{6.692\dots }\approx 2.587$ $\text{SNR}(0.001)=\sqrt{\frac{0.001}{0.999}}=\sqrt{0.001001}\approx 0.03164$ Early on the signal is ~2.6× louder than noise (digit recognisable); near the end noise is ~30× louder (essentially pure noise). This is exactly the parent note's MNIST intuition.

WHAT: use $x_0=(x_t-\sqrt{1-{\bar{\alpha }}_t}{ϵ}_{\theta })/\sqrt{{\bar{\alpha }}_t}$. $x_0=\frac{x_t-0.8ϵ}{0.6}=\frac{(0.6x_0+0.8ϵ)-0.8ϵ}{0.6}=\frac{0.6x_0}{0.6}=x_0$ With a perfect noise prediction, the reconstruction is exact — a good sanity check that the algebra closes.

WHAT: evaluate the outer scale $1/\sqrt{{\alpha }_t}$ and the inner noise coefficient ${\beta }_t/\sqrt{1-{\bar{\alpha }}_t}$. $\frac{1}{\sqrt{0.95}}=\frac{1}{0.97468}\approx 1.0260$ $\frac{0.05}{\sqrt{1-0.60}}=\frac{0.05}{\sqrt{0.40}}=\frac{0.05}{0.63246}\approx 0.07906$ So ${\mu }_{\theta }\approx 1.0260(x_t-0.07906{ϵ}_{\theta })$ — matching the parent note's $\approx 1.026(x_t-0.079{ϵ}_{\theta })$. WHY: we subtract the estimated noise, then rescale by $1/\sqrt{{\alpha }_t}>1$ to restore the signal amplitude that the forward step shrank.

${\tilde{\beta }}_t=\frac{1-0.62}{1-0.60}\times 0.05=\frac{0.38}{0.40}\times 0.05=0.95\times 0.05=0.0475$ WHY $<{\beta }_t$: knowing $x_t$ (and implicitly $x_0$ via the network) removes some uncertainty, so the reverse conditional is tighter than the raw forward noise ${\beta }_t=0.05$. The sampled next state is $x_{t-1}={\mu }_{\theta }+\sqrt{0.0475}z$ with $z\sim \mathcal{N}(0,I)$.

$(\sqrt{{\bar{\alpha }}_t}{)}^2+(\sqrt{1-{\bar{\alpha }}_t}{)}^2={\bar{\alpha }}_t+(1-{\bar{\alpha }}_t)=1.$ WHY it matters: if $\text{Var}(x_0)=1$ and $ϵ$ is independent with $\text{Var}=1$, then $\text{Var}(x_t)={\bar{\alpha }}_t\cdot 1+(1-{\bar{\alpha }}_t)\cdot 1=1.$ So the total variance stays pinned at $1$ for all $t$ — this is exactly the reason we shrink by $\sqrt{{\alpha }_t}$ instead of just adding noise. Picture the two bars: their squared heights always sum to the same fixed total. This is the analysis-level payoff of the "$\sqrt{1-{\beta }_t}$ shrink" design.

(a) ${\alpha }_1={\alpha }_2={\alpha }_3=0.8$, so ${\bar{\alpha }}_3=0.8^3=0.512$. (b) Composing: $x_3=\sqrt{{\alpha }_3}{x}_2+\cdots =\sqrt{{\alpha }_3{\alpha }_2{\alpha }_1}{x}_0+(\text{combined noise})$. The $x_0$-coefficient squared is ${\alpha }_1{\alpha }_2{\alpha }_3=0.512={\bar{\alpha }}_3$. ✔ Same answer — this is the whole point of the reparameterisation trick: nested single steps collapse into one product. Direct coefficient: $\sqrt{{\bar{\alpha }}_3}=\sqrt{0.512}\approx 0.7155$, and the noise coefficient $\sqrt{1-0.512}=\sqrt{0.488}\approx 0.6986$.

WHAT: solve $(1-\beta {)}^{1000}\le 0.001$. Take logs: $1000\ln (1-\beta )\le \ln (0.001)$, so $\ln (1-\beta )\le \frac{-6.90776}{1000}=-0.0069078$. $1-\beta \le e^{-0.0069078}=0.993116\Rightarrow \beta \ge \mathrm{0.006884.}$ So $\beta \approx 0.0069$. WHY this is instructive: it shows a single tiny $\beta$ across 1000 steps already crushes the signal to $0.1\%$ — matching why real schedules end near ${\beta }_T\approx 0.02$ (even more aggressive) and ${\bar{\alpha }}_{1000}\approx 0.001$.

(a) Noise the data. $\sqrt{{\bar{\alpha }}_t}=\sqrt{0.64}=0.8$, $\sqrt{1-{\bar{\alpha }}_t}=\sqrt{0.36}=0.6$. $x_t=0.8(1.0)+0.6(0.5)=0.8+0.3=1.1$ (b) Loss. $\Vert 0.5-0.5{\Vert }^2=0$ — a perfectly trained net has zero loss on this sample. ✔ (c) Reverse mean. Inner coefficient $\frac{{\beta }_t}{\sqrt{1-{\bar{\alpha }}_t}}=\frac{0.1}{0.6}=0.16667$; outer scale $\frac{1}{\sqrt{0.9}}=1.05409$. ${\mu }_{\theta }=1.05409(1.1-0.16667\times 0.5)=1.05409(1.1-0.08333)=1.05409\times 1.01667\approx 1.07166$ (d) Sample. $x_{t-1}={\mu }_{\theta }+\sqrt{{\tilde{\beta }}_t}z=1.07166+\sqrt{0.08}\times 1.0=1.07166+0.28284\approx 1.35450$. WHY the wobble past 1.0: the reverse step is stochastic (except at $t=1$) — that injected $\sqrt{{\tilde{\beta }}_t}z$ term is what gives generation its diversity. On average, though, ${\mu }_{\theta }$ is pulling $x_t$ back toward the clean signal.

Route A — network form. From 5.1(c) with these same numbers, ${\mu }_{\theta }\approx 1.07166$. Route B — true posterior ${\tilde{\mu }}_t=\frac{\sqrt{{\bar{\alpha }}_{t-1}}{\beta }_t}{1-{\bar{\alpha }}_t}{x}_0+\frac{\sqrt{{\alpha }_t}(1-{\bar{\alpha }}_{t-1})}{1-{\bar{\alpha }}_t}{x}_t$. Coefficient of $x_0$: $\frac{\sqrt{0.71}\times 0.1}{0.36}=\frac{0.84261\times 0.1}{0.36}=\frac{0.084261}{0.36}=0.234058$. Coefficient of $x_t$: $\frac{\sqrt{0.9}\times (1-0.71)}{0.36}=\frac{0.948683\times 0.29}{0.36}=\frac{0.275118}{0.36}=0.764216$. ${\tilde{\mu }}_t=0.234058(1.0)+0.764216(1.1)=0.234058+0.840637=1.074695$ The two routes agree to ~3 decimals ($1.0717$ vs $1.0747$; the tiny gap is because these hand-picked $\bar{\alpha }$ values are not perfectly self-consistent, i.e. ${\bar{\alpha }}_t\ne {\alpha }_t{\bar{\alpha }}_{t-1}$ exactly). WHY this matters: it confirms the noise-prediction parameterisation is not a hack — it reproduces the exact Bayes-optimal reverse mean when the network is perfect.

What always sums to 1 across the two coefficient bars? ::: ${\bar{\alpha }}_t+(1-{\bar{\alpha }}_t)=1$, keeping $\text{Var}(x_t)=1$. Why predict $ϵ$ not $x_0$? ::: noise is uniform across timesteps/domains, easier to learn; $x_0$ is recovered from it. Why is ${\tilde{\beta }}_t\le {\beta }_t$? ::: conditioning on $x_t$ removes uncertainty, tightening the reverse variance. Signal survival across steps is ______ ? ::: multiplicative — a product of ${\alpha }_t$, not a sum of ${\beta }_t$.