Latent diffusion (Stable Diffusion) — Hinglish notes

4.5.15 · AI-ML › Generative Models

Yeh Kaunsa Problem Solve Karta Hai?

Pixel-space diffusion ki problem: Ek 512×512 RGB image mein 786,432 dimensions hote hain. Is par kaafi saare timesteps ke liye U-Net denoising chalana computationally expensive hai, khaaskar self-attention layers.

Latent diffusion ka solution:

Images compress karne ke liye ek Variational Autoencoder (VAE) train karo: $x \in R^{H \times W \times 3} \to z \in R^{h \times w \times c}$ jahan $h, w ≪ H, W$
Diffusion sirf latent space $z$ mein chalao
Final $z_{0}$ ko VAE decoder se wapas image mein decode karo

Stable Diffusion $f = 8$ downsampling use karta hai: 512×512 → 64×64, $c = 4$ channels ke saath → 98% dimension reduction.

Architecture Components

1. The VAE (Autoencoder)

VAE kyun, aur simple autoencoder kyun nahi? VAE ka KL term latent space ko continuous aur smooth banane par majboor karta hai—nearby latents similar images mein decode hote hain. Diffusion ko noisy aur clean latents ke beech interpolate karne ke liye yeh smoothness chahiye.

Stable Diffusion specifics:

Downsampling factor $f = 8$ : 512→64, 1024→128
Latent channels $c = 4$ (learned, RGB nahi)
Massive image datasets (LAION) par alag se train kiya gaya
L2 pixel reconstruction loss + KL regularization ke saath train kiya gaya (released SD VAE yahi use karta hai; note: standard KL-autoencoder recipe mein VGG/perceptual loss nahi hota)
Diffusion training ke dauran frozen—sirf encode/decode ke liye use hota hai

2. The Diffusion U-Net (Latent Space Mein)

Pixels denoise karne ki jagah, hum latents $z_{t}$ denoise karte hain:

Reverse process (denoising):

Yeh kyun kaam karta hai:

$z$ 64×64×4 hai 512×512×3 ki jagah → U-Net ~2% original dimensions par operate karta hai
Latents semantic structure capture karte hain, isliye U-Net "object shapes" seekhta hai na ki "pixel noise patterns"
Text conditioning $c$ (CLIP se) denoising guide karta hai: "fur texture add karo" vs "metal texture add karo"

3. Text Conditioning (CLIP Embeddings)

Example: Prompt "a cat wearing sunglasses"

Head area ke corresponding latent region ka "cat" + "sunglasses" tokens par attention
Body ke latent region ka "cat" token par attention
U-Net is information ko use karta hai denoising ke dauran appropriate features inject karne ke liye

Figure — Latent diffusion (Stable Diffusion)

Step-by-Step: Stable Diffusion Image Kaise Generate Karta Hai

Worked example Generation Pipeline

Input: Text prompt "a corgi in a spacesuit" Step 1: CLIP se text encode karo $c = CLIPEncoder ("a corgi in a spacesuit")$

Step 2: Random latent noise sample karo $z_{T} \sim N (0, I), z_{T} \in R^{64 \times 64 \times 4}$

Step 3: $T$ steps ke liye denoise karo (jaise 50 steps) $t = T, T - 1, \dots, 1$ ke liye:

\hat{\epsilon} &= \epsilon_\theta(z_t, t, c) \quad \text{(U-Net se noise predict karo)}\\ z_{t-1} &= \frac{1}{\sqrt{\alpha_t}}\left[\, z_t - \frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}}\,\hat{\epsilon} \,\right] + \sigma_t \mathbf{z} \end{aligned}$$ jahan $\mathbf{z} \sim \mathcal{N}(0,I)$ naya noise hai (DDPM sampling), $\sigma_t$ stochasticity control karta hai, aur $\alpha_t = 1-\beta_t$. **Yeh step kyun?** Hum noise addition ko reverse kar rahe hain. $\frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}}\hat{\epsilon}$ term **predicted noise component remove karta hai**, leading $\frac{1}{\sqrt{\alpha_t}}$ factor poore bracket ko rescale karta hai, aur $\sigma_t\mathbf{z}$ diversity ke liye controlled randomness add karta hai. **Step 4:** Latent ko image mein decode karo $$x = \mathcal{D}(z_0) \quad \text{(VAE decoder)}$$ **Output:** 512×512 RGB image of corgi in spacesuit.

Latent Space Kyun? First Principles Se Derivation

Question: Pixel space se latent space mein denoise karna zyada efficient kyun hai?

Computational cost per denoising step (schematically): $cost \propto (number of spatial locations) \times (channels) \times (per-token work)$ aur self-attention part specifically spatial locations ki sankhya $N = h \cdot w$ mein quadratically scale karta hai: $attention cost \propto N^{2} = (h w)^{2}$

Pixel-space ke liye (DDPM):

Spatial locations: $512 \times 512 = 262, 144$
Channels: 3 (RGB)
Total dim: $262, 144 \times 3 = 786, 432$

Latent-space ke liye (Stable Diffusion):

Spatial locations: $64 \times 64 = 4, 096$
Channels: 4
Total dim: $4, 096 \times 4 = 16, 384$

Dimension ratio: $\frac{786 , 432}{16 , 384} = 48 \times$ kam dimensions.

Kyunki attention cost $N^{2}$ scale karta hai, spatial locations mein $262, 144$ se $4, 096$ tak ki reduction ( $N$ mein $64 \times$ drop) roughly yeh attention-cost reduction deta hai: $\frac{N _{pix}^{2}}{N _{lat}^{2}} = (\frac{262 , 144}{4 , 096})^{2} = 6 4^{2} = 4096 \times$ jabki element-wise convolutional work linearly scale karta hai $N$ ke saath (~ $64 \times$ cheaper). Practical end-to-end speedup is baat par depend karta hai ki U-Net kitna attention-heavy hai; empirically yeh large (roughly an order of magnitude ya usse zyada) hai, aur yahi reason hai ki latent diffusion ne consumer GPUs par high-res text-to-image generation feasible banaya.

Lekin quality loss kyun nahi hota?

VAE ko images reconstruct karne ke liye train kiya jaata hai (L2 loss) KL regularization ke saath. Yeh imperceptible high-frequency detail aur spatial redundancy (neighboring pixels ka almost identical hona) discard karta hai jabki semantic content—object shapes, colors, composition—preserve karta hai. Generative tasks ke liye hum semantic correctness care karte hain ("kya yeh cat jaisa dikhta hai?"), exact pixel values nahi, isliye yeh compression essentially perceptually harmless hai.

Common mistake Common Misconception: "Latent diffusion lower quality ka hota hai"

Galat idea: "64×64 mein compress karna detail lose karega—pixel diffusion sharper honi chahiye."

Yeh kyun sahi lagta hai: Compression = information loss, sahi? Aur 98% dimension reduction extreme lagta hai.

Fix: VAE lossy lekin perceptually near-lossless hai. Yeh imperceptible high-frequency noise aur spatial redundancy discard karta hai. 64×64×4 latent phir bhi saara semantic content encode karta hai—object shapes, textures, colors, composition.

Yeh kyun hold karta hai: Latent diffusion practically high resolution par pixel-space DDPM ko match ya beat karta hai, kyunki:

Latents par training bahut sasta hai → zyada training iterations → better convergence
Latent space mein better inductive bias hai (semantic structure, pixel noise nahi)
Compute budget dimensionality se ladne ki bajaye larger models aur datasets mein jaata hai

Common mistake Ek Aur Galti: "Main koi bhi autoencoder use kar sakta hun"

Galat idea: "Main ek simple convolutional autoencoder train karunga, phir uske latent space mein diffusion karunga."

Yeh kyun fail hota hai: Non-regularized autoencoders discontinuous latent spaces create karte hain "holes" ke saath—aise regions jo garbage mein decode hote hain. Agar aapka diffusion process denoising ke dauran kisi hole se guzre, toh image break ho jaati hai.

KL term kyun matter karta hai: KL divergence $D_{K L} (q (z ∣ x) ∥ p (z))$ latent distribution ko ek smooth, Gaussian-like prior ki taraf push karta hai. Iska matlab hai:

Do latents ke beech interpolation sensible images deta hai (koi holes nahi)
Noise add karna (diffusion forward process) valid latent regions mein rehta hai
Denoising (reverse process) manifold se "fall off" nahi karega

Steel-man the mistake: Aap soch sakte hain "KL divergence sirf ek regularizer hai, speed ke liye skip kar sakta hun." Lekin diffusion zaroor latent space mein continuously traverse karta hai—smoothness optional nahi hai.

Key Hyperparameters aur Design Choices

Parameter	Stable Diffusion v1.5	Yeh Value Kyun?
Latent channels $c$	4	Balance: 3 undercomplete hai (info lose karta hai), 8 redundant hai. 4 structure achha capture karta hai.
Downsampling $f$	8	$f = 4$ → abhi bhi large latents (slow), $f = 16$ → bahut lossy (VAE artifacts). 8 sweet spot hai.
U-Net timesteps $T$	1000 (training)	Standard DDPM schedule. Higher $T$ = smoother noise schedule lekin slower training.
Sampling steps	20–50 (inference)	DDIM/other samplers steps skip karte hain. ~50 steps ≈ near-max quality (iske aage diminishing returns).
CFG scale	7.5	Classifier-Free Guidance scale. Higher = zyada prompt adherence, lower = zyada diversity.

Classifier-Free Guidance (CFG): Sampling ke dauran, hum conditional aur unconditional predictions ke beech interpolate karte hain: $\overset{ϵ}{^} = ϵ_{θ} (z_{t}, t, \emptyset) + s \cdot (ϵ_{θ} (z_{t}, t, c) - ϵ_{θ} (z_{t}, t, \emptyset))$ jahan $s$ guidance scale hai (7.5), $\emptyset$ null prompt hai. Yeh text conditioning ko amplify karta hai.

Worked example Detailed Worked Example: Latent Sasta Kyun Hai

Setup: 512×512 image generate karo.

Pixel-space diffusion (DDPM):

Number of spatial tokens: $N_{pix} = 512 \times 512 = 262, 144$
In tokens par self-attention cost karta hai $\propto N_{pix}^{2} \approx 6.9 \times 1 0^{10}$ (relative units)

Latent diffusion (Stable Diffusion):

Ek baar encode karo: $E (x)$ 512→64 map karta hai
Number of spatial tokens: $N_{lat} = 64 \times 64 = 4, 096$
Self-attention cost karta hai $\propto N_{lat}^{2} \approx 1.7 \times 1 0^{7}$ (relative units)
Ek baar decode karo: $D (z_{0})$ 64→512 map karta hai

Attention-cost ratio: $\frac{N _{pix}^{2}}{N _{lat}^{2}} = (\frac{262 , 144}{4 , 096})^{2} = 6 4^{2} = 4096 \times$ attention layers mein sasta.

Yeh step kyun? Attention har token ko har doosre token ke saath compare karta hai, isliye spatial resolution half/eighth karne par cost quadratically kam hoti hai. Convolutional layers linearly scale karte hain (~ $64 \times$ ), aur VAE encode/decode sirf ek baar hota hai (har step mein nahi), isliye unka cost 20–50 denoising steps par negligible hai. Net effect: latent diffusion consumer GPUs par high-resolution generation practical banata hai (seconds mein, na ki woh impractically lamba time jo pixel-space attention 512² par require karta).

Recall Ek 12 Saal Ke Bacche Ko Samjhao

Imagine karo tum ek super detailed tasveer banana chahte ho, lekin tumhare paas sirf ek chhota notepad hai. Seedha saari details draw karna forever le jaata!

Yeh hai woh trick jo Stable Diffusion use karta hai:

Tasveer shrink karo – Apni badi tasveer lo aur use ek chhote "summary" mein compress karo (jaise apne phone par photo shrink karna). Yeh summary saari important cheezein rakhti hai—cat ka face, colors—lekin chhoti details jaise individual baal throw away kar deti hai.
Chhoti jagah mein draw karo – Ab ek bade canvas par draw karne ki jagah, tum is chhoti summary par draw karte ho. Bahut faster hai! Tum cat sketch kar sakte ho, sunglasses add kar sakte ho, galtiyan theek kar sakte ho—sab kuch super quick.
Wapas bada karo – Jab ho jao, chhoti summary ko wapas ek badi, detailed tasveer mein "un-compress" karo. Magic!

Yeh kyun kaam karta hai? Kyunki kisi bhi tasveer ka zyaatar hissa repetitive hota hai—blue sky pixels sab similar hain, grass sirf repeated texture hai. "Summary" unique parts rakhti hai (cat ki aankhein, sunglasses shape) aur baad mein repetitive parts recreate karne deti hai. Tum tasveer ki meaning par kaam kar rahe ho, na ki har ek pixel par.

Connections

Variational Autoencoders (VAE) – compression backbone
Denoising Diffusion Probabilistic Models (DDPM) – latent diffusion DDPM ka noise schedule latent space mein apply karta hai
U-Net Architecture – denoising model (conditioning ke liye attention layers ke saath)
CLIP Text Encoder – prompts ko cross-attention ke liye embeddings mein convert karta hai
Classifier-Free Guidance – text conditioning strengthen karne ki technique (woh CFG scale slider)
DDIM Sampling – faster sampling algorithm (isliye 20–50 steps, 1000 nahi)
Attention Mechanisms – text image generation ko kaise condition karta hai
Perceptual Loss – related idea; note karo ki SD KL-autoencoder L2+KL use karta hai, VGG nahi

#flashcards/ai-ml

Latent diffusion models ki key innovation kya hai? :: Diffusion process (noise addition aur denoising) ko ek low-dimensional latent space mein chalana jo VAE se produce hota hai, seedhe pixel space mein nahi, isse large speedup milta hai negligible quality loss ke saath.

Latent diffusion ke liye VAE (with KL) kyun use karte hain, plain autoencoder kyun nahi?

KL term latent distribution ko ek smooth Gaussian-like prior ki taraf push karta hai, "holes" prevent karta hai jahan interpolation break ho jaata hai. Diffusion ko latent space mein continuously traverse karna padta hai—smoothness ke bina, denoising artifacts produce karta hai.

Stable Diffusion v1.5 mein dimension reduction kitna hai?

512×512×3 (786,432 dimensions) → 64×64×4 (16,384 dimensions), f=8 downsampling VAE ke zariye ~98% reduction.

Latent diffusion forward process equation kya hai?

z_{t} = \overset{α}{ˉ}_{t} z_{0} + 1 - \overset{α}{ˉ}_{t} ϵ

jahan

z_{0} = E (x)

encoded image hai aur

ϵ \sim N (0, I)

noise hai.

Latent diffusion ke liye training objective kya hai? :: $L = E_{z_{0}, t, ϵ, c} [∥ ϵ - ϵ_{θ} (z_{t}, t, c) ∥^{2}]$ — U-Net ko train karo noise $ϵ$ predict karne ke liye diye gaye noisy latent $z_{t}$ , timestep $t$ , aur text conditioning $c$ ke liye.

DDPM reverse-step update kya hai?

z_{t - 1} = \frac{1}{α _{t}} [z_{t} - \frac{1 - α _{t}}{1 - α ˉ _{t}} \overset{ϵ}{^}] + σ_{t} z

, jahan

α_{t} = 1 - β_{t}

aur

z \sim N (0, I)

Stable Diffusion mein text conditioning kaise kaam karta hai?

Text CLIP se encode hokar embedding

c

banta hai, phir U-Net mein cross-attention ke zariye inject hota hai. Query image features se aata hai, key/value text se, jisse har spatial region relevant prompt tokens par attend kar sake.

Stable Diffusion VAE ko kaunsa loss train karta hai?

Pixel-level L2 reconstruction loss + ek KL divergence regularization term (standard KL-autoencoder recipe); us recipe mein VGG/perceptual loss nahi hota.

Self-attention pixel-space diffusion ko itna expensive kyun banata hai?

Attention cost spatial tokens ki sankhya

N = h w

mein

N^{2}

scale karta hai. 512² par 262,144 tokens hain; 64² par sirf 4,096, isliye attention cost

(262144/4096)^{2} = 4096 \times

drop ho jaata hai.

Stable Diffusion generation ke chaar steps kya hain?

1) CLIP se text encode karo →

c

, 2) Noise sample karo

z_{T} \sim N (0, I)

, 3) U-Net

ϵ_{θ} (z_{t}, t, c)

use karke T steps denoise karo, 4) VAE decoder

D (z_{0})

z_{0}

ko image mein decode karo.

Classifier-Free Guidance (CFG) kya hai?

\overset{ϵ}{^} = ϵ_{θ} (z_{t}, t, \emptyset) + s (ϵ_{θ} (z_{t}, t, c) - ϵ_{θ} (z_{t}, t, \emptyset))

. Scale

s

prompt adherence vs diversity control karta hai.

SD VAE space mein diffusion latent manifold se "fall off" kyun nahi karta?

KL regularization latents ko near-Gaussian aur smooth banata hai, isliye saare points valid images mein interpolate hote hain—denoising ke dauran koi "holes" nahi hote jisme girna pade.