Denoising diffusion probabilistic models (DDPM)

4.5.11 · AI-ML › Generative Models

Overview

Denoising Diffusion Probabilistic Models (DDPMs) generative models hain jo data create karna seekhte hain ek gradual noising process ko reverse karke. Data distribution ko directly seekhne ki jagah, yeh models ek Markov chain ke har step par denoising seekhte hain, jisse learning problem tractable ban jaati hai.

Yeh kyun kaam karta hai: Chhote denoising steps Gaussian distributions ko approximate karte hain (Central Limit Theorem ki wajah se), jinhe hum neural networks se model karna jaante hain.

The Forward Diffusion Process

Yeh Kya Karta Hai

Forward process dheere dheere data mein Gaussian noise add karta hai $T$ timesteps ke upar, structure ko tabah karta hai jab tak sirf pure noise nahi bach jaati.

$q({\mathbf{x}}_t|{\mathbf{x}}_{t-1})=\mathcal{N}({\mathbf{x}}_t;\sqrt{1-{\beta }_t}{\mathbf{x}}_{t-1},{\beta }_t\mathbf{I})$

jahaan ${\beta }_1,{\beta }_2,..,{\beta }_T$ ek variance schedule hai jisme $0<{\beta }_t<1$.

Yeh Form Kyun?

$\sqrt{1-{\beta }_t}$ se multiply kyun karte hain? Total variance ko control karne ke liye. Agar hum sirf noise $\mathcal{N}(0,{\beta }_t\mathbf{I})$ add karte, toh variance unboundedly badhta: $\text{Var}[{\mathbf{x}}_t]=\text{Var}[{\mathbf{x}}_{t-1}]+{\beta }_t$. Scaling factor variance ko stable rakhta hai.

Derivation from scratch: ${\mathbf{x}}_{t-1}$ ko variance ${\sigma }_{t-1}^2$ ke saath shuru karte hue: ${\mathbf{x}}_t=\sqrt{1-{\beta }_t}{\mathbf{x}}_{t-1}+\sqrt{{\beta }_t}\mathbf{ϵ},\mathbf{ϵ}\sim \mathcal{N}(0,\mathbf{I})$

Phir: $\text{Var}[{\mathbf{x}}_t]=(1-{\beta }_t){\sigma }_{t-1}^2+{\beta }_t$

Agar hum chahte hain ${\sigma }_t^2={\sigma }_{t-1}^2=1$ (unit variance), toh initial data normalized hona chahiye. Yeh property math ko enormously simplify kar deti hai.

$q({\mathbf{x}}_t|{\mathbf{x}}_0)=\mathcal{N}({\mathbf{x}}_t;\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0,(1-{\bar{\alpha }}_t)\mathbf{I})$

Yeh powerful kyun hai: Hum ${\mathbf{x}}_t$ ko KISI BHI timestep par directly ${\mathbf{x}}_0$ se sample kar sakte hain, bina saare intermediate steps iterate kiye!

Derivation: ${\mathbf{x}}_1=\sqrt{{\alpha }_1}{\mathbf{x}}_0+\sqrt{1-{\alpha }_1}{\mathbf{ϵ}}_1$ ${\mathbf{x}}_2=\sqrt{{\alpha }_2}{\mathbf{x}}_1+\sqrt{1-{\alpha }_2}{\mathbf{ϵ}}_2$

${\mathbf{x}}_1$ substitute karte hue: ${\mathbf{x}}_2=\sqrt{{\alpha }_2\_1}{\mathbf{x}}_0+\sqrt{{\alpha }_2(1-{\alpha }_1)}{\mathbf{ϵ}}_1+\sqrt{1-{\alpha }_2}{\mathbf{ϵ}}_2$

Kyunki dono noise terms Gaussian hain, yeh combine ho jaate hain: $\sqrt{{\alpha }_2(1-{\alpha }_1)}{\mathbf{ϵ}}_1+\sqrt{1-{\alpha }_2}{\mathbf{ϵ}}_2=\sqrt{{\alpha }_2(1-{\alpha }_1)+(1-{\alpha }_2)}\mathbf{ϵ}$

Simplify karte hue: ${\alpha }_2(1-{\alpha }_1)+(1-{\alpha }_2)={\alpha }_2-{\alpha }_2{\alpha }_1+1-{\alpha }_2=1-{\alpha }_1{\alpha }_2$

$t$ steps ke liye induction se: ${\mathbf{x}}_t=\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0+\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ}$.

The Reverse Process (Generation)

Hum Kya Chahte Hain

Data generate karne ke liye, humein diffusion ko reverse karna hai: noise ${\mathbf{x}}_T\sim \mathcal{N}(0,\mathbf{I})$ se shuru karke dheere dheere ${\mathbf{x}}_0$ tak denoise karna hai.

Saccha reverse process $q({\mathbf{x}}_{t-1}|{\mathbf{x}}_t)$ intractable hai kyunki iske liye poori data distribution jaanna zaroori hai. Lekin ${\mathbf{x}}_0$ par conditioned hone par, reverse step Gaussian hoti hai!

$q({\mathbf{x}}_{t-1}|{\mathbf{x}}_t,{\mathbf{x}}_0)=\mathcal{N}({\mathbf{x}}_{t-1};{\tilde{\mathbf{\mu }}}_t({\mathbf{x}}_t,{\mathbf{x}}_0),{\tilde{\beta }}_t\mathbf{I})$

jahaan: ${\tilde{\mathbf{\mu }}}_t({\mathbf{x}}_t,{\mathbf{x}}_0)=\frac{\sqrt{{\bar{\alpha }}_{t-1}}{\beta }_t}{1-{\bar{\alpha }}_t}{\mathbf{x}}_0+\frac{\sqrt{{\alpha }_t}(1-{\bar{\alpha }}_{t-1})}{1-{\bar{\alpha }}_t}{\mathbf{x}}_t$

${\tilde{\beta }}_t=\frac{1-{\bar{\alpha }}_{t-1}}{1-{\bar{\alpha }}_t}{\beta }_t$

Yeh form kyun? Yeh do Gaussians ko multiply karke aur square complete karke aata hai.

Derivation from Bayes: $q({\mathbf{x}}_{t-1}|{\mathbf{x}}_t,{\mathbf{x}}_0)\propto q({\mathbf{x}}_t|{\mathbf{x}}_{t-1})q({\mathbf{x}}_{t-1}|{\mathbf{x}}_0)$

Dono Gaussians hain. $\mathcal{N}(\mathbf{x};\mu ,{\sigma }^2)$ ke liye, log density hai $-\frac{1}{2{\sigma }^2}(\mathbf{x}-\mu {)}^2+\text{const}$.

Expand karte hue: $\log q({\mathbf{x}}_t|{\mathbf{x}}_{t-1})=-\frac{1}{2{\beta }_t}({\mathbf{x}}_t-\sqrt{{\alpha }_t}{\mathbf{x}}_{t-1}{)}^2+C_1$

$\log q({\mathbf{x}}_{t-1}|{\mathbf{x}}_0)=-\frac{1}{2(1-{\bar{\alpha }}_{t-1})}({\mathbf{x}}_{t-1}-\sqrt{{\bar{\alpha }}_{t-1}}{\mathbf{x}}_0{)}^2+C_2$

Sum karo aur ${\mathbf{x}}_{t-1}^2$ terms collect karo variance ke liye, linear terms mean ke liye. Algebra se ${\tilde{\beta }}_t$ aur ${\tilde{\mathbf{\mu }}}_t$ milta hai.

Learned Reverse Process

Hum reverse ko ek neural network se approximate karte hain:

$p_{\theta }({\mathbf{x}}_{t-1}|{\mathbf{x}}_t)=\mathcal{N}({\mathbf{x}}_{t-1};{\mathbf{\mu }}_{\theta }({\mathbf{x}}_t,t),{\Sigma }_{\theta }({\mathbf{x}}_t,t))$

Training Objective

Variational Lower Bound

Negative log-likelihood ka ek variational lower bound hai:

$-\log p_{\theta }({\mathbf{x}}_0)\le \mathbb{E}\left[-\log \frac{p_{\theta }({\mathbf{x}}_{0:T})}{q({\mathbf{x}}_{1:T}|{\mathbf{x}}_0)}\right]=L$

Yeh in terms mein decompose hota hai:

$L={\mathbb{E}}_q[D_{KL}(q({\mathbf{x}}_T|{\mathbf{x}}_0)\Vert p({\mathbf{x}}_T))$ $+{\sum }_{t=2}^T{D}_{KL}(q({\mathbf{x}}_{t-1}|{\mathbf{x}}_t,{\mathbf{x}}_0)\Vert p_{\theta }({\mathbf{x}}_{t-1}|{\mathbf{x}}_t))$ $-\log p_{\theta }({\mathbf{x}}_0|{\mathbf{x}}_1)]$

Yeh terms kyun?

• Pehla term: Hamari final noise standard Gaussian se kitni door hai? (Koi parameters nahi, ignore karo)
• Middle sum: Hamara learned reverse, true reverse se kitna match karta hai?
• Aakhri term: Reconstruction likelihood

$L_{\text{simple}}={\mathbb{E}}_{t,{\mathbf{x}}_0,\mathbf{ϵ}}\left[\Vert \mathbf{ϵ}-{\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t){\Vert }^2\right]$

jahaan ${\mathbf{x}}_t=\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0+\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ}$ aur $t\sim \text{Uniform}(1,T)$.

Mean ki jagah noise predict kyun karte hain? Kyunki hum reparameterize kar sakte hain: ${\mathbf{\mu }}_{\theta }({\mathbf{x}}_t,t)=\frac{1}{\sqrt{{\alpha }_t}}\left({\mathbf{x}}_t-\frac{{\beta }_t}{\sqrt{1-{\bar{\alpha }}_t}}{\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t)\right)$

Noise $\mathbf{ϵ}$ predict karna empirically ${\mathbf{x}}_0$ ya $\mathbf{\mu }$ directly predict karne se better kaam karta hai—yeh timesteps ke across ek zyada stable learning target hai.

Reparameterization ki derivation: ${\mathbf{x}}_t=\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0+\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ}$ se, ${\mathbf{x}}_0$ ke liye solve karo: ${\mathbf{x}}_0=\frac{{\mathbf{x}}_t-\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ}}{\sqrt{{\bar{\alpha }}_t}}$

${\tilde{\mathbf{\mu }}}_t$ formula mein substitute karo aur ${\bar{\alpha }}_t={\alpha }_t{\bar{\alpha }}_{t-1}$ use karke simplify karo upar wali form paane ke liye.

Step 1: Ek training image ${\mathbf{x}}_0$ sample karo (jaise ek billi ki photo, $[-1,1]$ tak normalized).

Step 2: Ek random timestep $t=437$ uniformly sample karo.

Step 3: Noise $\mathbf{ϵ}\sim \mathcal{N}(0,\mathbf{I})$ image ke same shape mein sample karo.

Step 4: ${\bar{\alpha }}_{437}={\prod }_{s=1}^{437}(1-{\beta }_s)\approx 0.542$ compute karo (precomputed).

Step 5: Noised image banao: ${\mathbf{x}}_{437}=\sqrt{0.542}\cdot {\mathbf{x}}_0+\sqrt{1-0.542}\cdot \mathbf{ϵ}$

Step 6: ${\mathbf{x}}_{437}$ aur $t=437$ ko network mein feed karo, prediction ${\mathbf{ϵ}}_{\theta }({\mathbf{x}}_{437},437)$ lo.

Step 7: Loss compute karo: $L=\Vert \mathbf{ϵ}-{\mathbf{ϵ}}_{\theta }({\mathbf{x}}_{437},437){\Vert }^2$, backpropagate karo.

Yeh step kyun? $t=437$ par, image moderately noised hai. Network seekhta hai: "Diye hue specific noise level aur partially corrupted image ke saath, exactly woh noise predict karo jo add ki gayi thi." Bahut saare examples mein, bahut saare timesteps par, yeh full denoising distribution seekh leta hai.

Sampling (Generation)

Step 1: ${\mathbf{x}}_T\sim \mathcal{N}(0,\mathbf{I})$ sample karo (pure noise, jaise $256\times 256\times 3$ random numbers ka tensor).

Step 2: $t=T,T-1,...,1$ ke liye:

• Noise predict karo: ${\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t)$
• Mean compute karo: ${\mathbf{\mu }}_{\theta }=\frac{1}{\sqrt{{\alpha }_t}}\left({\mathbf{x}}_t-\frac{{\beta }_t}{\sqrt{1-{\bar{\alpha }}_t}}{\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t)\right)$
• Noise $\mathbf{z}\sim \mathcal{N}(0,\mathbf{I})$ sample karo agar $t>1$, warna $\mathbf{z}=0$
• Update karo: ${\mathbf{x}}_{t-1}={\mathbf{\mu }}_{\theta }+\sqrt{{\beta }_t}\mathbf{z}$

Step 3: ${\mathbf{x}}_0$ return karo (generated image).

Har step mein noise kyun? Reverse process stochastic hai—hum ek distribution se sample kar rahe hain, koi deterministic function compute nahi kar rahe. Added noise $\mathbf{z}$ reverse step mein uncertainty account karta hai.

$t=1$ par noise kyun nahi? Final step mein, hum $p_{\theta }({\mathbf{x}}_0|{\mathbf{x}}_1)$ ka mode (most likely value) chahte hain, jo mean hai.

Key Design Choices

Variance Schedule

Common choices:

• Linear: ${\beta }_t={\beta }_1+\frac{t-1}{T-1}({\beta }_T-{\beta }_1)$
• Cosine: ${\bar{\alpha }}_t=\frac{f(t)}{f(0)}$ jahaan $f(t)=\cos {\left(\frac{t/T+s}{1+s}\cdot \frac{\pi }{2}\right)}^2$

Network Architecture

Typically ek U-Net hota hai jisme:

• Timestep $t$ sinusoidal positional encoding ke roop mein embed hota hai
• Skip connections dono high aur low-level features pass karne ke liye
• Global structure capture karne ke liye Self-attention layers

U-Net kyun? Denoising task ke liye dono local (texture) aur global (object shape) context chahiye. U-Net ki architecture ismein excel karti hai.

DDPMs ke Advantages

1. High-quality samples: State-of-art image generation quality (GANs ke saath competitive FID scores)
2. Stable training: Koi adversarial dynamics nahi, sirf noise prediction par regression
3. Flexible: Condition karna aasaan (class labels, text, etc.) ${\mathbf{ϵ}}_{\theta }$ mein conditioning add karke
4. Probabilistic: Explicit likelihood bound, GANs ke unlike

Class labels ke saath train karo. Generation time par:

• Class $c$ sample karo (ya choose karo)
• ${\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t,c)$ ke saath sampling algorithm chalao

Result: Specified class ki images. Yeh text conditioning tak extend hota hai (CLIP ke zariye text embeddings) text-to-image models jaise Stable Diffusion ke liye.

Common Mistakes

Steel-man: Chhoti noise ke liye, single-step denoising (jaise denoising autoencoders) theek kaam karta hai. "Simpler is better" ki intuition generally sound hai.

Yeh galat kyun hai: Pure noise → structured image ek bahut hi complex mapping hai. Distribution $p({\mathbf{x}}_0|{\mathbf{x}}_T)$ multimodal, sharply peaked structure rakhti hai—single neural network pass ke liye intractable. $T$ chhote steps mein todne se, har step ek Gaussian approximate karta hai (unimodal, smooth), jise networks aasaani se model kar sakti hain.

Fix: Iterative nature accept karo. Bahut saare steps hi problem ko learnable banate hain. (Modern research better schedules aur distillation se steps kam karta hai, lekin fundamentally multiple steps zaroori hain.)

Yeh galat kyun hai: Ek bura schedule (jaise bahut jaldi zyada noise) early timesteps ko unlearnable bana deta hai—network signal ko noise se distinguish nahi kar sakta. Bahut slow schedule redundant easy denoising steps par capacity waste karta hai.

Fix: Proven schedules use karo (images ke liye cosine). Schedule ek hyperparameter hai jo model ke learning ki difficulties ka curriculum shape karta hai.

Yeh galat kyun hai: Empirically, $\mathbf{ϵ}$ predict karna faster convergence aur better sample quality deta hai. Reason: $\mathbf{ϵ}$ timesteps ke across ek zyada balanced target hai (hamesha same distribution), jabki high noise levels par ${\mathbf{x}}_0$ prediction poorly constrained hoti hai.

Fix: Noise prediction parameterization use karo (ya velocity prediction, ek newer variant). Architectural choices ke liye empirical evidence ko guide karne do.

Connections to Other Concepts

• Score-based generative models: DDPMs score matching ke equivalent hain Langevin dynamics ke saath; ${\mathbf{ϵ}}_{\theta }$ score ${\nabla }_{\mathbf{x}}\log p({\mathbf{x}}_t)$ approximate karta hai
• Variational Autoencoders (VAE): DDPMs similar ELBO optimize karte hain, lekin fixed encoder (forward process) ke saath
• Stochastic Differential Equations: DDPMs ka continuous-time limit ek reverse-time SDE deta hai
• Markov Chain Monte Carlo: Sampling iterative refinement hai, MC jaisi, lekin learned transitions ke saath
• U-Net architecture: Image DDPMs mein ${\mathbf{ϵ}}_{\theta }$ ka backbone
• Classifier-free guidance: Conditional aur unconditional predictions mix karke conditional generation strengthen karne ki technique
• Latent diffusion models: Efficiency ke liye compressed latent space (VAE encoder) mein diffusion chalao—Stable Diffusion ka basis

Ya yaad rakho: "Forward fuzes, backward builds"—forward process images ko noise mein fuse karta hai, backward unhe wapas build karta hai.

Ab, agar tumhare paas ek jaadu ka tool hota jo raat ko ek second ek time par, reverse mein, hata sakta? Second 999 par, yeh thodi si raat hataata hai aur tumhe rang ki ek chhoti si jhalak dikhti hai. Second 500 par, tum shapes dekhna shuru kar sakte ho. Second 1 tak, painting phir se saaf hai.

Yahi ek diffusion model karta hai! Computer us jaadu ke tool banna seekhta hai. Hum usse bahut saari paintings dikhate hain jo raat se dhaaki ja rahi hain (yeh forward process hai—noise add karna). Phir hum use unhe step by step un-cover karna sikhate hain (yeh reverse process hai—noise remove karna). Ek baar jab yeh seekh jaata hai, hum pure raat (random noise) se shuru kar sakte hain aur ise ek aisi painting "uncover" karne de sakte hain jo pehle kabhi thi hi nahi—yeh destruction process ko reverse karke naya art create kar raha hai!

Clever part yeh hai: ek baar mein saari raat hatana seekhna bahut mushkil hai. Lekin thodi thodi raat ek baar mein hatana? Woh bahut aasaan hai, jaise ek baar mein ek speck of dust saaf karna rather than poora ghar saaf karna. Toh hum problem ko 1000 chhote chhote cleaning steps mein tod dete hain, aur computer har ek mein bahut achha ho jaata hai.

Practice Questions

#flashcards/ai-ml

DDPM mein forward diffusion process kya hai? :: Ek Markov chain jo dheere dheere data mein T timesteps ke upar Gaussian noise add karta hai: $q({\mathbf{x}}_t|{\mathbf{x}}_{t-1})=\mathcal{N}({\mathbf{x}}_t;\sqrt{1-{\beta }_t}{\mathbf{x}}_{t-1},{\beta }_t\mathbf{I})$, structure ko tabah karta hai jab tak sirf noise nahi bach jaati.

${\mathbf{x}}_0$ se directly ${\mathbf{x}}_t$ sample karne ka closed-form kya hai? :: $q({\mathbf{x}}_t|{\mathbf{x}}_0)=\mathcal{N}({\mathbf{x}}_t;\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0,(1-{\bar{\alpha }}_t)\mathbf{I})$ jahaan ${\bar{\alpha }}_t={\prod }_{s=1}^t(1-{\beta }_s)$. Yeh bina iterate kiye kisi bhi timestep par jump karne deta hai.

DDPM ka simplified training objective kya hai? :: $L_{\text{simple}}={\mathbb{E}}_{t,{\mathbf{x}}_0,\mathbf{ϵ}}\left[\Vert \mathbf{ϵ}-{\mathbf{ϵ}}_{\theta }({\mathbf{x}}_t,t){\Vert }^2\right]$ jahaan $t\sim \text{Uniform}(1,T)$ aur ${\mathbf{x}}_t=\sqrt{{\bar{\alpha }}_t}{\mathbf{x}}_0+\sqrt{1-{\bar{\alpha }}_t}\mathbf{ϵ}$.

Variance schedule $\{{\beta }_t\}$ ka kya role hai? :: Yeh control karta hai ki forward process mein noise kitni jaldi add hoti hai. Ek achha schedule timesteps ke across learning difficulty balance karta hai—bahut jaldi early steps ko unlearnable banata hai, bahut dheere capacity easy steps par waste karta hai. Common: linear ya cosine schedules.

Concept Map