Noise scheduling

4.5.12 · AI-ML › Generative Models

Noise Scheduling Kya Hai?

YEH KYUN MATTER KARTA HAI? Schedule decide karta hai:

1. Training stability: Bure schedules → gradient vanishing/exploding
2. Sample quality: Kharab schedules → artifacts, mode collapse
3. Speed: Achhe samples ke liye kitne steps T chahiye

Schedule Mathematics Ko Derive Karna

Single-Step Se Multi-Step Tak

Recurrence se shuru karo (kyun: humein $x_t$ ko $x_0$ se relate karna hai):

$x_t=\sqrt{{\alpha }_t}{x}_{t-1}+\sqrt{1-{\alpha }_t}{ϵ}_{t-1},{ϵ}_{t-1}\sim \mathcal{N}(0,I)$

Recursively expand karo: $x_t=\sqrt{{\alpha }_t}(\sqrt{{\alpha }_{t-1}}{x}_{t-2}+\sqrt{1-{\alpha }_{t-1}}{ϵ}_{t-2})+\sqrt{1-{\alpha }_t}{ϵ}_{t-1}$

Key insight: Independent Gaussians ka sum. Agar $Z_1\sim \mathcal{N}(0,{\sigma }_1^2)$ aur $Z_2\sim \mathcal{N}(0,{\sigma }_2^2)$, tab: $Z_1+Z_2\sim \mathcal{N}(0,{\sigma }_1^2+{\sigma }_2^2)$

Apply karo: $x_t=\sqrt{{\alpha }_t{\alpha }_{t-1}}{x}_{t-2}+\sqrt{{\alpha }_t(1-{\alpha }_{t-1})+(1-{\alpha }_t)}ϵ$

Variance simplify karo (kyun yeh step: noise terms combine karne ke liye algebra): ${\alpha }_t(1-{\alpha }_{t-1})+(1-{\alpha }_t)={\alpha }_t-{\alpha }_t{\alpha }_{t-1}+1-{\alpha }_t=1-{\alpha }_t{\alpha }_{t-1}$

$x_0$ tak continue karo: $x_t=\sqrt{{\bar{\alpha }}_t}{x}_0+\sqrt{1-{\bar{\alpha }}_t}ϵ,{\bar{\alpha }}_t={\prod }_{s=1}^t{\alpha }_s$

${\bar{\alpha }}_t$ kyun matter karta hai: Ek single number jo step $t$ pe bachi hui total signal capture karta hai.

Common Schedules

Design Principles

1. Signal-to-Noise Ratio (SNR)

Step $t$ pe SNR define karo: $\text{SNR}(t)=\frac{{\bar{\alpha }}_t}{1-{\bar{\alpha }}_t}$

Derivation:

• Signal component ka variance: ${\bar{\alpha }}_t\Vert x_0{\Vert }^2$
• Noise ka variance: $(1-{\bar{\alpha }}_t)d$ (dimension $d$)
• Ratio se SNR milta hai

WHY it matters: Neural network tab best seekhta hai jab SNR smoothly change ho (≈ har step pe constant difficulty).

2. Boundary Conditions

$t=0$ pe (start):

• Chahte hain ${\bar{\alpha }}_0\approx 1$ (almost no noise added); SNR parametrization mein hum ise $1-\epsilon$ pe cap karte hain taaki log-SNR finite rahe
• ${\beta }_1$ bahut chhota hona chahiye par non-zero (numerical stability)

$t=T$ pe (end):

• ${\bar{\alpha }}_T\approx 0$ chahiye (prior $\mathcal{N}(0,I)$ se indistinguishable)
• Typically ${\bar{\alpha }}_T<10^{-2}$ aim karo

WHY: Kharab boundaries → network ko edges pe zyada mehnat karni padti hai, artifacts aate hain.

Adaptive Schedules

Connections

• 4.5.11-Forward-and-reverse-diffusion-process — Schedule forward process $q(x_t|x_{t-1})$ define karta hai
• 4.5.13-Denoising-objective — Schedule loss weighting across timesteps determine karta hai
• 4.5.15-Score-matching — SNR score function magnitude ${\nabla }_{x_t}\log p(x_t)$ ko affect karta hai
• 4.5.18-Fast-sampling — Schedule choice step-skipping strategies (DDIM) enable karta hai

#flashcards/ai-ml

Concept Map