Exercises — Noise scheduling

What we do: apply the definition ${\alpha }_t=1-{\beta }_t$. ${\alpha }_5=1-0.01=0.99$ Meaning: at step $5$, a single noising step keeps $99\%$ of the signal amplitude and swaps in $1\%$ worth of noise. It is a per-step survival fraction, not cumulative.

What we do: ${\bar{\alpha }}_t$ is the product of all $\alpha$ up to $t$, because signal survival compounds — each step multiplies what is left. ${\bar{\alpha }}_3=0.99\times 0.98\times 0.97=0.941094$ So after three steps about $94.1\%$ of the original signal amplitude remains.

Recall the symbols: $q(x_t\mid x_0)$ is the shortcut distribution for the noised image at step $t$ given the clean $x_0$; its mean is $\sqrt{{\bar{\alpha }}_t}{x}_0$ and its variance is $(1-{\bar{\alpha }}_t)$ per pixel (the $I$ means "same in every dimension"). Mean coefficient = $\sqrt{{\bar{\alpha }}_t}=\sqrt{0.25}=0.5$. The clean image is scaled to half its brightness. Variance = $1-{\bar{\alpha }}_t=1-0.25=0.75$. Why variance = energy: for a zero-mean random pixel, its variance is its average squared value $\mathbb{E}[{\text{noise}}^2]$ — i.e. its mean power (energy per pixel). So a variance of $0.75$ literally says three-quarters of each pixel's energy is now pure Gaussian fuzz.

Why this formula is "linear": we want ${\beta }_t$ to rise from ${\beta }_1$ to ${\beta }_T$ in equal increments as $t$ marches $1\to T$ — a straight line in $t$ — because that is the simplest interpolation that guarantees the noise added grows steadily and ${\bar{\alpha }}_T$ reaches near-zero. The factor $\frac{t-1}{T-1}$ is just "how far along the line we are," from $0$ at $t=1$ to $1$ at $t=T$. What we do: the step size in $\beta$ is $\frac{{\beta }_T-{\beta }_1}{T-1}=\frac{0.4-0.1}{3}=0.1$.

• ${\beta }_1=0.1$
• ${\beta }_2=0.1+0.1=0.2$
• ${\beta }_3=0.1+0.2=0.3$
• ${\beta }_4=0.1+0.3=0.4$ Evenly spaced, exactly as "linear" promises.

What we do: turn each $\beta$ into $\alpha =1-\beta$, then multiply (signal survival compounds step by step, hence a product). ${\alpha }_1,{\alpha }_2,{\alpha }_3,{\alpha }_4=0.9,0.8,0.7,0.6$ ${\bar{\alpha }}_4=0.9\times 0.8\times 0.7\times 0.6=0.3024$ Only about $30\%$ of the signal survives after $4$ aggressive steps.

What we do: apply $\text{SNR}(t)=\frac{{\bar{\alpha }}_t}{1-{\bar{\alpha }}_t}$. $\text{SNR}=\frac{0.3024}{1-0.3024}=\frac{0.3024}{0.6976}=0.433486\dots$ $\log \text{SNR}=\ln (0.433486)=-0.835773\dots$ Negative log-SNR means noise now dominates signal (ratio below $1$).

Why this tool (adding Gaussian variances): ${ϵ}_{t-2}$ and ${ϵ}_{t-1}$ are independent standard normals. For independent Gaussians, variances add — that is the single fact that lets us merge two noise terms into one. What we do: collect the two noise coefficients.

• Coefficient of ${ϵ}_{t-2}$ is $\sqrt{{\alpha }_t}\sqrt{1-{\alpha }_{t-1}}$, so its variance contribution is ${\alpha }_t(1-{\alpha }_{t-1})$.
• Coefficient of ${ϵ}_{t-1}$ is $\sqrt{1-{\alpha }_t}$, contributing $(1-{\alpha }_t)$. Add them: ${\alpha }_t(1-{\alpha }_{t-1})+(1-{\alpha }_t)={\alpha }_t-{\alpha }_t{\alpha }_{t-1}+1-{\alpha }_t=1-{\alpha }_t{\alpha }_{t-1}.$ The ${\alpha }_t$ terms cancel, leaving one clean noise term $\mathcal{N}(0,1-{\alpha }_t{\alpha }_{t-1})$. Iterating this is exactly how the parent note reaches $x_t=\sqrt{{\bar{\alpha }}_t}{x}_0+\sqrt{1-{\bar{\alpha }}_t}ϵ$.

Figure below (s01): two black bell curves — the solid one is the noise from step $t-1$ (variance ${\alpha }_t(1-{\alpha }_{t-1})$), the dashed one is the noise from step $t$ (variance $1-{\alpha }_t$). The red curve is their merged single Gaussian, whose variance is the sum $1-{\alpha }_t{\alpha }_{t-1}$ — notice the red curve is wider (more spread) than either black one, because adding variances always widens.

What we do: apply $\log \frac{\bar{\alpha }}{1-\bar{\alpha }}$ to each.

• Linear: $\frac{0.03}{0.97}=0.030928$, $\log =-3.47610\dots$
• Cosine: $\frac{0.50}{0.50}=1$, $\log =0$. Meaning: at the exact midpoint the linear schedule is already deep in noise (log-SNR $\approx -3.5$), so half of its remaining timesteps denoise near-pure static — wasted effort; cosine sits at the balanced point (log-SNR $=0$), spreading difficulty evenly.

What we do: start from the definition and solve for ${\bar{\alpha }}_t$. $e^{\lambda }=\frac{{\bar{\alpha }}_t}{1-{\bar{\alpha }}_t}$ Multiply both sides by $(1-{\bar{\alpha }}_t)$: $e^{\lambda }(1-{\bar{\alpha }}_t)={\bar{\alpha }}_t\Rightarrow e^{\lambda }={\bar{\alpha }}_t+e^{\lambda }{\bar{\alpha }}_t={\bar{\alpha }}_t(1+e^{\lambda })$ ${\bar{\alpha }}_t=\frac{e^{\lambda }}{1+e^{\lambda }}=\sigma (\lambda ).$ This is why an SNR-parametrised schedule is just "push a straight line for $\lambda (t)$ through a sigmoid."

Why finite ${\lambda }_{\max }$: the true start has ${\bar{\alpha }}_0=1$ (empty product — see the definition callout up top), giving $\text{SNR}(0)=\frac{1}{1-1}=\infty$ and ${\lambda }_0=+\infty$, which is undefined. Capping at ${\lambda }_{\max }=2$ makes ${\bar{\alpha }}_0$ slightly below $1$, the tiny-noise-at-start behaviour we want. What we do: slope $=\frac{{\lambda }_{\max }-{\lambda }_{\min }}{T}=\frac{2-(-2)}{4}=1$ per step, decreasing.

• $\lambda (0)=2\Rightarrow {\bar{\alpha }}_0=\sigma (2)=\frac{e^2}{1+e^2}=0.880797$
• $\lambda (2)=2-2\cdot 1=0\Rightarrow {\bar{\alpha }}_2=\sigma (0)=0.5$
• $\lambda (4)=2-4\cdot 1=-2\Rightarrow {\bar{\alpha }}_4=\sigma (-2)=0.119203$ Signal fraction glides $0.881\to 0.5\to 0.119$ — smooth and, crucially, finite at both ends.

Figure below (s02): the red curve is $\bar{\alpha }(t)=\sigma (\lambda (t))$, the cumulative signal, gliding smoothly from $\approx 0.88$ down through $0.5$ to $\approx 0.12$; the dashed black line is the straight log-SNR $\lambda (t)$ (rescaled to fit the same axes) that we pushed through the sigmoid. Red dots mark the three computed points at $t=0,2,4$ — the S-shape of the red curve is precisely the "slow–fast–slow" decay the sigmoid produces from a straight line.

What we do: first get ${\bar{\alpha }}_1=\sigma (1)=\frac{e^1}{1+e^1}=0.731059$. Then ${\beta }_2=1-\frac{{\bar{\alpha }}_2}{{\bar{\alpha }}_1}=1-\frac{0.5}{0.731059}=1-0.683940=\mathrm{0.316060.}$ So the SNR design implies ${\beta }_2\approx 0.316$ — you never chose $\beta$ directly; it fell out of the log-SNR line. This is the whole point of SNR-first design.

Speedup: $\frac{1000}{250}=4\times$ fewer sampling steps. FID change: $3.21-3.17=0.04$ higher FID (lower FID is better, so this is a tiny quality loss). Verdict & why: $4\times$ faster for a $0.04$ FID penalty is an excellent trade. The reason cosine survives step-skipping is that its log-SNR decreases smoothly and near-uniformly across time — each retained step still carries roughly equal information, so dropping $75\%$ of them does not create a "difficulty cliff." A linear schedule dumps most of its useful work into a narrow early band and would degrade far more if thinned. This connects directly to 4.5.18-Fast-sampling.

Start condition: need $\sigma ({\lambda }_{\max })\ge 1-10^{-3}=0.999$. Solve $\sigma (\lambda )=0.999\Rightarrow \lambda =\log \frac{0.999}{0.001}=\log (999)=6.90675\dots$ Smallest integer ${\lambda }_{\max }$ meeting this is $\boxed{7}$ (since $\sigma (7)=0.999089>0.999$, while $\sigma (6)=0.997527<0.999$). End condition: ${\bar{\alpha }}_T=\sigma (-6)=\frac{e^{-6}}{1+e^{-6}}=0.00247262<0.01$. ✓ Satisfied. Both boundaries pass, so the certified line is $\lambda (t)=7-\frac{t}{1000}(7-(-6))=7-0.013t$.

Starting point: by the empty-product convention ${\bar{\alpha }}_0=1$ (stated in the definition callout at the top). Strictly decreasing: ${\bar{\alpha }}_t={\bar{\alpha }}_{t-1}\cdot {\alpha }_t$. Since $0<{\alpha }_t<1$, multiplying by ${\alpha }_t$ shrinks a positive number: ${\bar{\alpha }}_t<{\bar{\alpha }}_{t-1}$. So the sequence falls at every step. Bounded in $(0,1)$: it is a product of factors each in $(0,1)$, so it stays positive and below $1$ (from ${\bar{\alpha }}_0=1$, every later term is strictly below $1$). Guarantee: a strictly decreasing positive sequence converges. Choosing ${\alpha }_t$ so the product drives ${\bar{\alpha }}_T\to 0$ means $q(x_T\mid x_0)=\mathcal{N}(\sqrt{{\bar{\alpha }}_T}{x}_0,(1-{\bar{\alpha }}_T)I)\to \mathcal{N}(0,I)$ — the endpoint becomes the fixed prior, independent of $x_0$. That $x_0$-independence is exactly what lets sampling start from pure noise, the foundation of 4.5.11-Forward-and-reverse-diffusion-process.

Products not sums (${\bar{\alpha }}_t=\prod {\alpha }_s$), and ${\bar{\alpha }}_0=1$ by empty product ::: L1–L2 core skill Independent Gaussian variances add, not std-devs ::: L3 core skill ${\bar{\alpha }}_t=\sigma (\lambda )$ inverts log-SNR ::: L3–L4 bridge Finite ${\lambda }_{\max }$ keeps the start well-defined ::: L4 boundary insight FID is a distance (lower = better); smooth log-SNR enables step-skipping ::: L5 judgement