Noise scheduling

What Is Noise Scheduling?

WHY does this matter? The schedule determines:

Training stability: Bad schedules → gradient vanishing/exploding
Sample quality: Poor schedules → artifacts, mode collapse
Speed: Number of steps T needed for good samples

Deriving the Schedule Mathematics

From Single-Step to Multi-Step

Start with the recurrence (why: we need to relate $x_t$ to $x_0$ ):

$x_t = \sqrt{\alpha_t} x_{t-1} + \sqrt{1-\alpha_t} \epsilon_{t-1}, \quad \epsilon_{t-1} \sim \mathcal{N}(0, I)$

Expand recursively: $x_t = \sqrt{\alpha_t}(\sqrt{\alpha_{t-1}} x_{t-2} + \sqrt{1-\alpha_{t-1}} \epsilon_{t-2}) + \sqrt{1-\alpha_t} \epsilon_{t-1}$

Key insight: Sum of independent Gaussians. If $Z_1 \sim \mathcal{N}(0, \sigma_1^2)$ and $Z_2 \sim \mathcal{N}(0, \sigma_2^2)$ , then: $Z_1 + Z_2 \sim \mathcal{N}(0, \sigma_1^2 + \sigma_2^2)$

Apply: $x_t = \sqrt{\alpha_t \alpha_{t-1}} x_{t-2} + \sqrt{\alpha_t(1-\alpha_{t-1}) + (1-\alpha_t)} \epsilon$

Simplify the variance (why this step: algebra to combine noise terms): $\alpha_t(1-\alpha_{t-1}) + (1-\alpha_t) = \alpha_t - \alpha_t\alpha_{t-1} + 1 - \alpha_t = 1 - \alpha_t\alpha_{t-1}$

Continue to $x_0$ : $x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1-\bar{\alpha}_t} \epsilon, \quad \bar{\alpha}_t = \prod_{s=1}^t \alpha_s$

WHY $\bar{\alpha}_t$ matters: Single number that captures total signal remaining at step $t$ .

Common Schedules

Worked example Numerical Comparison (T=1000, computed by cumulative product)

Linear Schedule ( $\beta_1=10^{-4}$ , $\beta_T=0.02$ ):

$t=100$ : $\bar{\alpha}_{100} \approx 0.90$ (about 10% signal lost)
$t=500$ : $\bar{\alpha}_{500} \approx 0.03$ (signal almost gone)
$t=900$ : $\bar{\alpha}_{900} \approx 5\times10^{-5}$ (essentially pure noise) Cosine Schedule ( $s=0.008$ ):
$t=100$ : $\bar{\alpha}_{100} \approx 0.98$ (slower decay, signal preserved)
$t=500$ : $\bar{\alpha}_{500} \approx 0.50$ (still half signal at midpoint)
$t=900$ : $\bar{\alpha}_{900} \approx 0.02$ (approaching noise, but gently)

Why this difference: Linear rushes early and collapses the signal well before $t=T$ , wasting many late steps on near-pure noise. Cosine keeps the signal alive longer and distributes the destruction more evenly across time.

Design Principles

1. Signal-to-Noise Ratio (SNR)

Define SNR at step $t$ : $\text{SNR}(t) = \frac{\bar{\alpha}_t}{1-\bar{\alpha}_t}$

Derivation:

Variance of signal component: $\bar{\alpha}_t \|x_0\|^2$
Variance of noise: $(1-\bar{\alpha}_t) d$ (dimension $d$ )
Ratio gives SNR

WHY it matters: Neural network learns best when SNR changes smoothly (≈ constant difficulty per step).

Worked example SNR-Based Schedule Design

Goal: Make $\log \text{SNR}(t)$ decrease linearly over the interior of the schedule.

Important boundary caveat: At $t=0$ we have $\bar{\alpha}_0 = 1$ , so $\text{SNR}(0) = \frac{1}{1-1} = \infty$ — the log-SNR is $+\infty$ and cannot be a finite $\lambda_0$ . We must regularize: either start the schedule at a small $t_{\min} > 0$ , or clip $\bar{\alpha}_0$ to $1 - \varepsilon$ so that $\lambda_{\max} = \log\frac{1-\varepsilon}{\varepsilon}$ is finite.

Regularized construction: pick finite endpoints $\lambda_{\max}$ (start) and $\lambda_{\min}$ (end), and interpolate linearly: $\lambda(t) = \lambda_{\max} - \frac{t}{T}(\lambda_{\max} - \lambda_{\min}), \qquad \lambda(t) := \log\text{SNR}(t)$

Solve for $\bar{\alpha}_t$ (invert the SNR definition): $\frac{\bar{\alpha}_t}{1-\bar{\alpha}_t} = e^{\lambda(t)} \;\Rightarrow\; \bar{\alpha}_t = \frac{e^{\lambda(t)}}{1 + e^{\lambda(t)}} = \sigma(\lambda(t))$ where $\sigma$ is the sigmoid. Because $\lambda_{\max}$ is finite (not $+\infty$ ), $\bar{\alpha}_0 = \sigma(\lambda_{\max})$ is slightly below 1 — exactly the tiny-noise-at-start behavior we want.

Why this works: Uniform (log-SNR) difficulty for the denoising network across all timesteps, without an ill-defined infinite initial SNR.

2. Boundary Conditions

At $t=0$ (start):

Want $\bar{\alpha}_0 \approx 1$ (almost no noise added); in an SNR parametrization we cap it at $1-\varepsilon$ to keep log-SNR finite
$\beta_1$ should be very small but non-zero (numerical stability)

At $t=T$ (end):

Need $\bar{\alpha}_T \approx 0$ (indistinguishable from prior $\mathcal{N}(0,I)$ )
Typically aim for $\bar{\alpha}_T < 10^{-2}$

WHY: Poor boundaries → network must work harder at edges, artifacts appear.

Adaptive Schedules

Connections

4.5.11-Forward-and-reverse-diffusion-process — Schedule defines forward process $q(x_t|x_{t-1})$
4.5.13-Denoising-objective — Schedule determines loss weighting across timesteps
4.5.15-Score-matching — SNR affects score function magnitude $\nabla_{x_t} \log p(x_t)$
4.5.18-Fast-sampling — Schedule choice enables step-skipping strategies (DDIM)

Recall Explain to a 12-year-old

Imagine you're trying to guess what a picture looks like, but someone keeps adding more and more blur to it. The noise schedule is like the recipe for how much blur to add each second.

If you add tons of blur right away, you'll lose important details like faces or text — game over. If you add blur super slowly, it'll take forever to blur it completely. The best schedules (like "cosine") add blur slowly at first (keeping details), speed up in the middle, then slow down again at the end.

Why? Because our "de-blurring robot" (the neural network) learns better when each step is equally hard. If one step is too easy and another is impossible, the robot gets confused. A good schedule makes every step just-right difficult!

#flashcards/ai-ml

What is the mathematical relationship between

\beta_t

\alpha_t

, and

\bar{\alpha}_t

in noise scheduling?

\alpha_t = 1 - \beta_t

(per-step signal retention),

\bar{\alpha}_t = \prod_{s=1}^t \alpha_s

(cumulative signal retention from

x_0

x_t

)

What is the correct linear schedule formula?

\beta_t = \beta_1 + \frac{(t-1)(\beta_T - \beta_1)}{T-1}

— a linear interpolation from

\beta_1

(typically

10^{-4}

) to

\beta_T

(typically

0.02

)

Why does cosine schedule outperform linear schedule in diffusion models?

Cosine provides S-curve decay: preserves signal longer initially (fewer early artifacts), smoother SNR gradient (stable training), avoids the linear schedule's early collapse where signal is nearly gone by the midpoint, wasting late steps

Derive the direct sampling formula

x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1-\bar{\alpha}_t} \epsilon

Start with

x_t = \sqrt{\alpha_t} x_{t-1} + \sqrt{1-\alpha_t}\epsilon_{t-1}

. Expand recursively, use Gaussian sum property

\mathcal{N}(0,\sigma_1^2) + \mathcal{N}(0,\sigma_2^2) = \mathcal{N}(0,\sigma_1^2+\sigma_2^2)

, simplify variances to get

\sqrt{\bar{\alpha}_t}

coefficient on

x_0

What is the SNR at timestep

t

and why does it matter for schedule design?

\text{SNR}(t) = \bar{\alpha}_t/(1-\bar{\alpha}_t)

. Measures signal-to-noise ratio; constant

\log \text{SNR}

decay ensures uniform difficulty per denoising step, leading to stable gradients and better convergence

Why is

\log\text{SNR}(0)

ill-defined and how do we fix it in an SNR-based schedule?

t=0

\bar{\alpha}_0=1

\text{SNR}(0)=1/0=\infty

and

\log\text{SNR}=+\infty

. Fix by regularizing: cap

\bar{\alpha}_0

1-\varepsilon

(giving finite

\lambda_{\max}=\log\frac{1-\varepsilon}{\varepsilon}

) or start the schedule at

t_{\min}>0

Why must

\beta_1 > 0

(not exactly zero) even though we want to preserve information initially?

Exactly zero causes: (1) division by

\sqrt{\alpha_1}=1

numerical issues, (2) zero gradient flow (no learning signal), (3) infinite/undefined log-SNR since

\bar{\alpha}_1=1

. Tiny

\beta_1\approx 10^{-4}

provides stability without significant information loss

What boundary conditions must a good noise schedule satisfy?

t=0

\bar{\alpha}_0 \approx 1

(no noise; capped at

1-\varepsilon

for finite log-SNR). At

t=T

\bar{\alpha}_T < 0.01

(nearly pure noise, matches prior

\mathcal{N}(0,I)

). Poor boundaries cause edge artifacts

How is

\beta_t

computed from a

\bar{\alpha}_t

schedule?

From definition

\bar{\alpha}_t = \bar{\alpha}_{t-1} \alpha_t = \bar{\alpha}_{t-1}(1-\beta_t)

, rearrange:

\beta_t = 1 - \bar{\alpha}_t/\bar{\alpha}_{t-1}

. Clamp to valid range

[10^{-4}, 0.999]

for numerical stability

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, diffusion models ka basic idea simple hai: hum ek clean image mein thoda-thoda karke noise add karte hain jab tak woh pure random noise na ban jaye, aur phir model ko sikhate hain ki is process ko ulta kaise kare. Yahan noise scheduling ka matlab hai ki har step par kitna noise add karna hai - ye ek recipe hai. Socho tum ek photo ko acid mein dhire-dhire ghula rahe ho: agar shuru mein hi bahut tezi se ghula doge to important details permanently lost ho jayengi aur model reverse nahi kar payega; agar bahut slowly karoge to time waste hoga. Isliye schedule ( $\beta_t$ ya $\alpha_t$ values) ko carefully balance karna padta hai.

Ab maths ki taraf - ek badi khubsurat baat ye hai ki independent Gaussian noises ko add karne se variance simply add ho jaata hai. Isi property ki wajah se hum recurrence ko expand karke ek direct formula nikaal sakte hain: $x_t = \sqrt{\bar{\alpha}_t}\, x_0 + \sqrt{1-\bar{\alpha}_t}\, \epsilon$ . Yahan $\bar{\alpha}_t$ (cumulative product of $\alpha$ values) ek single number hai jo batata hai step $t$ par original signal ka kitna hissa bacha hai. Iska practical fayda ye hai ki hume step-by-step noise add nahi karna padta - kisi bhi timestep ka noisy image ek hi shot mein sample kar sakte hain, jo training ko fast banata hai.

Ab linear vs cosine schedule ka farak samajhna zaroori hai. Linear schedule shuru mein hi signal ko jaldi khatam kar deta hai - $t=500$ tak toh signal almost gaya, matlab baaki ke late steps kaafi waste ho jaate hain kyunki wahan already pure noise hai. Cosine schedule zyada smart hai: ye signal ko lambe time tak zinda rakhta hai (S-curve shape mein - slow start, fast middle, slow end), destruction ko evenly distribute karta hai. Isse gradients stable rehte hain, artifacts kam aate hain aur sample quality behtar hoti hai. Ye matter karta hai kyunki galat schedule se tumhara model ya toh train hi nahi hoga (vanishing/exploding gradients) ya phir ghatiya, blurry images banayega - toh scheduling ek chhoti si cheez lagti hai par poore model ki performance decide karti hai.

What Is Noise Scheduling?

Deriving the Schedule Mathematics

From Single-Step to Multi-Step

Common Schedules

Design Principles

1. Signal-to-Noise Ratio (SNR)

2. Boundary Conditions

Adaptive Schedules

Connections

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Generative Models

Connections