Variational Autoencoders (VAE) theory — Hinglish notes

4.5.3 · AI-ML › Generative Models

#ai-ml/generative-models #unsupervised-learning #probabilistic-models #latent-variables

Overview

Variational Autoencoders (VAEs) generative models hain jo data ko ek latent space mein encode karna aur wapas decode karna seekhte hain, saath hi yeh ensure karte hain ki latent space ek jaani-pehchaani distribution (typically Gaussian) follow kare. Standard autoencoders ke unlike, VAEs probabilistic hote hain aur naye data ko sample karne ki suvidha dete hain.

Core Intuition

Hume probabilistic formulation ki zaroorat kyun hai?

Deterministic autoencoders latent space mein "memorize" kar sakte hain aur disconnected regions bana sakte hain
Hum latent space se sample karke naye samples generate karna chahte hain
Probabilistic interpretation hume reconstruction aur regularization ke beech balance karne ka ek principled tarika deta hai

Mathematical Foundation

The Generative Story

Gaussian prior kyun? Yeh mathematically convenient hai, infinite support hai, aur central limit theorem suggest karta hai ki bahut saari natural processes iske taraf converge hoti hain.

Problem: Intractable Marginal Likelihood

Hum maximize karna chahte hain: $p_{θ} (x) = \int p_{θ} (x ∣ z) p (z) d z$

Yeh mushkil kyun hai? Har data point $x$ ke liye, hume ALL possible latent codes $z$ par integrate karna padega yeh dekhne ke liye ki kaun se ise generate kar sakte the. High-dimensional $z$ ke saath, yeh integral computationally intractable hai.

Solution: Variational Inference

Ek inference network (encoder) $q_{ϕ} (z ∣ x)$ introduce karo jo true posterior $p_{θ} (z ∣ x)$ ko approximate karta hai.

ELBO ko scratch se derive karna:

Log-likelihood se shuru karo: $lo g p_{θ} (x) = lo g \int p_{θ} (x ∣ z) p (z) d z$

Andar $\frac{q _{ϕ} ( z ∣ x )}{q _{ϕ} ( z ∣ x )} = 1$ se multiply karo: $lo g p_{θ} (x) = lo g \int q_{ϕ} (z ∣ x) \frac{p _{θ} ( x ∣ z ) p ( z )}{q _{ϕ} ( z ∣ x )} d z$

Yeh $q_{ϕ}$ par expectation hai: $lo g p_{θ} (x) = lo g E_{q_{ϕ} (z ∣ x)} [\frac{p _{θ} ( x ∣ z ) p ( z )}{q _{ϕ} ( z ∣ x )}]$

Yeh step kyun? Jensen's inequality kehti hai ki $lo g E [X] \geq E [lo g X]$ kisi bhi random variable $X$ ke liye (log concave hota hai). Ise apply karo:

$lo g p_{θ} (x) \geq E_{q_{ϕ} (z ∣ x)} [lo g \frac{p _{θ} ( x ∣ z ) p ( z )}{q _{ϕ} ( z ∣ x )}]$

Logarithm ko split karo: $lo g p_{θ} (x) \geq E_{q_{ϕ} (z ∣ x)} [lo g p_{θ} (x ∣ z)] + E_{q_{ϕ} (z ∣ x)} [lo g \frac{p ( z )}{q _{ϕ} ( z ∣ x )}]$

Doosra term negative KL divergence hai: $E_{q_{ϕ} (z ∣ x)} [lo g \frac{p ( z )}{q _{ϕ} ( z ∣ x )}] = - D_{K L} (q_{ϕ} (z ∣ x) ∥ p (z))$

"Lower bound" kyun? Jensen se aaya inequality guarantee karta hai ki $lo g p_{θ} (x) \geq L$ . ELBO maximize karna log-likelihood par ek lower bound ko upar dhakelta hai.

Gap: Approximate aur True Posterior ke Beech KL

Hum dikha sakte hain: $lo g p_{θ} (x) = L (θ, ϕ; x) + D_{K L} (q_{ϕ} (z ∣ x) ∥ p_{θ} (z ∣ x))$

Kyun? Yahan se shuru karo: $D_{K L} (q_{ϕ} ∥ p_{θ} (z ∣ x)) = E_{q_{ϕ}} [lo g \frac{q _{ϕ} ( z ∣ x )}{p _{θ} ( z ∣ x )}]$

Bayes rule use karo: $p_{θ} (z ∣ x) = \frac{p _{θ} ( x ∣ z ) p ( z )}{p _{θ} ( x )}$ :

$D_{K L} (q_{ϕ} ∥ p_{θ} (z ∣ x)) = E_{q_{ϕ}} [lo g q_{ϕ} (z ∣ x) - lo g p_{θ} (x ∣ z) - lo g p (z) + lo g p_{θ} (x)]$

Kyunki $lo g p_{θ} (x)$ $z$ par depend nahi karta: $D_{K L} (q_{ϕ} ∥ p_{θ} (z ∣ x)) = lo g p_{θ} (x) - L (θ, ϕ; x)$

Kyunki KL divergence hamesha $\geq 0$ hoti hai, yeh confirm karta hai ki ELBO ek lower bound hai.

The Reparameterization Trick

Problem: Hume $E_{q_{ϕ} (z ∣ x)} [\cdot]$ ke through gradients compute karne hain, lekin sampling differentiable nahi hai.

Maano $q_{ϕ} (z ∣ x) = N (μ_{ϕ} (x), σ_{ϕ}^{2} (x))$ .

Standard sampling (non-differentiable): $z \sim N (μ_{ϕ} (x), σ_{ϕ}^{2} (x))$

Intuition: Socho jaise "decision" (mean aur variance, jo parameters par depend karte hain) ko "randomness" (epsilon, jo nahi karta) se alag karna. Ab gradients decision part ke through flow kar sakte hain.

Practical VAE Architecture

Log variance predict kyun karte hain? Positivity ensure karta hai ( $σ^{2} = exp (lo g σ^{2}) > 0$ ) aur numerical stability deta hai.

Practice Mein Loss Function

Dataset $D = {x^{(i)}}_{i = 1}^{N}$ ke liye:

$L (θ, ϕ) = \frac{1}{N} \sum_{i = 1}^{N} [E_{q_{ϕ} (z ∣ x^{(i)})} [lo g p_{θ} (x^{(i)} ∣ z)] - D_{K L} (q_{ϕ} (z ∣ x^{(i)}) ∥ p (z))]$

Monte Carlo estimate (typically 1 sample per data point): $L \approx \frac{1}{N} \sum_{i = 1}^{N} [lo g p_{θ} (x^{(i)} ∣ z^{(i)}) - D_{K L} (q_{ϕ} (z ∣ x^{(i)}) ∥ N (0, I))]$

jahan $z^{(i)} = μ_{ϕ} (x^{(i)}) + σ_{ϕ} (x^{(i)}) ⊙ ϵ^{(i)}$ .

Gaussians ke Liye KL Divergence (Closed Form)

$q = N (μ, σ^{2} I)$ aur $p = N (0, I)$ ke liye:

Derivation: $D_{K L} (q ∥ p) = E_{q} [lo g q (z)] - E_{q} [lo g p (z)]$

Multivariate Gaussian (dimension $d$ ) ke liye: $lo g q (z) = - \frac{d}{2} lo g (2 π) - \sum_{j = 1}^{d} lo g σ_{j} - \frac{1}{2} \sum_{j = 1}^{d} \frac{( z _{j} - μ _{j} ) ^{2}}{σ _{j}^{2}}$

$lo g p (z) = - \frac{d}{2} lo g (2 π) - \frac{1}{2} \sum_{j = 1}^{d} z_{j}^{2}$

$q$ par expectations lete hue (use karke $E_{q} [z_{j}] = μ_{j}$ , $E_{q} [z_{j}^{2}] = μ_{j}^{2} + σ_{j}^{2}$ ):

$E_{q} [lo g q (z)] = - \frac{d}{2} lo g (2 π) - \sum_{j = 1}^{d} lo g σ_{j} - \frac{d}{2}$

$E_{q} [lo g p (z)] = - \frac{d}{2} lo g (2 π) - \frac{1}{2} \sum_{j = 1}^{d} (μ_{j}^{2} + σ_{j}^{2})$

Subtract karo: $D_{K L} (q ∥ p) = - \sum_{j = 1}^{d} lo g σ_{j} - \frac{d}{2} + \frac{1}{2} \sum_{j = 1}^{d} (μ_{j}^{2} + σ_{j}^{2})$

$= \frac{1}{2} \sum_{j = 1}^{d} (μ_{j}^{2} + σ_{j}^{2} - lo g σ_{j}^{2} - 1)$

Worked Examples

Worked example Example 1: MNIST ke Liye ELBO Compute Karna

Setup: Binary MNIST images ( $28 \times 28$ ), latent dimension $d = 2$ .

Encoder outputs: $μ = [0.5, - 0.3]$ , $lo g σ^{2} = [- 1.0, - 0.5]$
Sample: $ϵ = [0.2, - 0.4]$
Decoder har pixel ke liye Bernoulli logits output karta hai

Step 1: Reparameterization ke zariye $z$ compute karo $σ_{1} = exp (- 1.0/2) = exp (- 0.5) \approx 0.606$ $σ_{2} = exp (- 0.5/2) = exp (- 0.25) \approx 0.779$ $z_{1} = 0.5 + 0.606 \times 0.2 = 0.621$ $z_{2} = - 0.3 + 0.779 \times (- 0.4) = - 0.612$

Yeh steps kyun? Hum differentiable sample paane ke liye reparameterization trick apply kar rahe hain.

Step 2: KL term compute karo $D_{K L} = \frac{1}{2} [(0. 5^{2} + 0.60 6^{2} - (- 1.0) - 1) + (- 0.3)^{2} + 0.77 9^{2} - (- 0.5) - 1)]$ $= \frac{1}{2} [(0.25 + 0.367 + 1.0 - 1) + (0.09 + 0.607 + 0.5 - 1)]$ $= \frac{1}{2} [0.617 + 0.197] = 0.407$

Kyun? Yeh measure karta hai ki encoder distribution standard normal se kitna deviate karta hai.

Step 3: Reconstruction term compute karo Decoder har pixel ke liye logits $ℓ_{ij}$ produce karta hai. Agar true pixel 1 hai: $lo g p_{θ} (x_{ij} = 1∣ z) = lo g σ (ℓ_{ij}) = - lo g (1 + e^{- ℓ_{ij}})$

Total reconstruction log-likelihood paane ke liye sabhi 784 pixels par sum karo.

Step 4: ELBO = Reconstruction - KL Maano reconstruction term = -150 (MNIST ke liye typical): $L = - 150 - 0.407 = - 150.407$

Negative kyun? Hum log-likelihood (probabilities < 1 ka log) compute kar rahe hain. Higher (kam negative) better hai.

Common Mistakes

Common mistake Mistake 1: Reparameterization Trick Bhool Jana

Galat approach:

z = np.random.normal(mu, sigma)  # Non-differentiable!
reconstruction = decoder(z)
loss.backward()  # Gradients don't flow to encoder

Yeh sahi kyun lagta hai: "Main woh distribution se sample kar raha hoon jo encoder output karta hai."

Yeh galat kyun hai: Sampling operation gradient flow tod deta hai. $\frac{\partial z}{\partial ϕ} = 0$ kyunki sampling ek stochastic operation hai.

Sahi approach:

epsilon = torch.randn_like(sigma)
z = mu + sigma * epsilon  # Differentiable!
reconstruction = decoder(z)
loss.backward()  # Gradients flow through mu and sigma

Fix: Randomness $ϵ$ mein (parameters se independent), $μ$ aur $σ$ ke saath deterministic transformation (parameters jinhein hum differentiate kar sakte hain).

Common mistake Mistake 2: Sirf Standard Autoencoder Loss Use Karna

Galat approach:

loss = reconstruction_loss(decoder(encoder(x)), x)

Yeh sahi kyun lagta hai: "Mujhe bas achhe reconstructions chahiye!"

Yeh galat kyun hai: KL regularization ke bina:

Encoder alag-alag inputs ko door-door, arbitrary codes mein map kar sakta hai
Latent space mein holes aur discontinuities hain
Naye samples generate nahi kar sakte (prior se sampling realistic data mein decode nahi hogi)
Overfitting: training data memorize kar leta hai

Sahi approach:

reconstruction_loss = ...
kl_loss = 0.5 * torch.sum(mu**2 + sigma**2 - torch.log(sigma**2) - 1)
loss = reconstruction_loss + kl_loss

Fix: KL term encoder ko prior ki structure use karne par majboor karta hai, generation enable karta hai.

Common mistake Mistake 3: KL Vanishing (Posterior Collapse)

Problem: Training ke dauran, KL term zero ho jata hai, encoder input ignore karta hai, decoder prior se "average" images generate karna seekh leta hai.

Yeh kyun hota hai:

Decoder bahut powerful hai (large capacity)
$z$ mein information ki zaroorat ke bina prior se seedha $p (x)$ model kar sakta hai
Encoder shortcut leta hai: sabhi inputs ke liye prior distribution $N (0, I)$ output karta hai Yeh pehle sahi kyun lagta hai: "Kam KL matlab prior se achha match!"

Yeh galat kyun hai: $x$ aur $z$ ke beech zero mutual information. Latent codes bekaar hain.

Fixes:

KL annealing: $β \cdot D_{K L}$ mein $β ≪ 1$ se shuru karo, gradually 1 tak badhao
Free bits: Sirf tab KL penalize karo jab woh ek threshold se neeche ho (kuch bits of information allow karo)
Weaker decoder: Capacity limit karo $z$ par reliance force karne ke liye

Intuition: Hum woh sweet spot dhundhna chahte hain jahan decoder ko $z$ ki zaroorat ho lekin $z$ phir bhi prior ke close rahe.

Active Recall Checks

Recall VAEs Ko Ek 12-Saal-Ke-Bachche Ko Explain Karo

Socho tumhare paas ek magic box hai jo cat ki photos ko secret codes mein compress kar sakta hai (sirf kuch numbers), aur ek aur magic box hai jo un codes ko wapas cat photos mein badal sakta hai.

Ek regular magic box thoda messy ho sakta hai—kuch codes bahut door hain, kuch codes bilkul kaam nahi karte. Agar tum ek random code try karo, tumhe garbage mil sakta hai.

Ek VAE ek zyada smart magic box hai jo:

Ensure karta hai ki saare secret codes ek pattern follow karein (jaise ek bell curve)
Ensure karta hai ki paas-paas codes similar cats dein
Koi empty spots nahi—har code jo tum try karo tumhe ek cat deta hai!

Trick yeh hai: Photos ko compress aur uncompress karna seekhte waqt, boxes codes ko nicely organize karna bhi seekhte hain. Ab tum:

Koi bhi random code do → ek naya cat photo pao jo real lagta hai
Do cat photos lo, unke codes mix karo, ek aisa cat pao jo beech-beech mein hai "Variational" part ka matlab hai hum probability aur smart math use kar rahe hain yeh ensure karne ke liye ki organization automatically ho jaye!

Memory Aids

Reparameterization: "Random ko learnable se alag karo"

Random: $ϵ$ (standard normal, koi parameters nahi)
Learnable: $μ$ aur $σ$ (encoder outputs)

KL Divergence Terms: "My Variance Logically Matters"

Mean squared: $μ^{2}$ (zero se door mean ki penalty)
Variance: $σ^{2}$ (zyada spread ki penalty)
Log variance: $- lo g σ^{2}$ (kam spread ki penalty)
Minus one: $- 1$ (normalization)

Connections

Autoencoders: VAEs autoencoders ko probabilistic formulation ke saath extend karte hain
Kullback-Leibler Divergence: Distribution mismatch measure karne ke liye core hai
Bayesian Inference: VAEs approximate posteriors ke liye variational inference use karte hain
Latent Variable Models: VAEs amortized inference ke saath latent variable models hain
Evidence Lower Bound (ELBO): Fundamental objective function
Reparameterization Trick: Gradient-based learning enable karta hai
Generative Adversarial Networks (GANs): Alternative generative model approach
Normalizing Flows: Probabilistic generative models ki ek aur family
Posterior Collapse: VAEs mein common training pathology
Disentangled Representations: Interpretability ke liye Beta-VAE aur doosre variants
Conditional VAE: Conditional generation ke liye extension
Importance-Weighted Autoencoders: Tighter ELBO bounds

#flashcards/ai-ml

Autoencoders aur VAEs mein fundamental difference kya hai? :: VAEs probabilistic hote hain (points ki jagah distributions encode/decode karte hain), inka latent space regularized hota hai (KL term prior enforce karta hai), aur prior se sampling karke generation enable karte hain.

ELBO ka full form kya hai aur iske do components kaun se hain?

Evidence Lower Bound. Components: (1) Reconstruction term - latent code diye gaye data ki expected log-likelihood, (2) KL divergence term - regularization jo posterior ko prior ke close rakhta hai.

Hume reparameterization trick ki zaroorat kyun hai?

Encoder distribution se standard sampling non-differentiable hai, encoder parameters mein gradient flow block karta hai. Reparameterization random component (ε) ko learnable parameters (μ, σ) se alag karta hai, sampling process ko differentiable banata hai.

Reparameterization trick ka formula likhо :: z = μ_φ(x) + σ_φ(x) ⊙ ε, jahan ε ~ N(0, I). Randomness ε mein hai (φ se independent), gradients ko μ aur σ ke through flow karne deta hai.

Posterior collapse kya hai aur yeh kyun hota hai? :: Jab KL term zero ho jata hai, encoder input ignore karta hai aur sabhi inputs ke liye prior distribution output karta hai. Hota hai jab decoder itna powerful hota hai ki bina z mein information ki zaroorat ke prior se seedha p(x) model kar sakta hai.

Diagonal Gaussian aur standard normal ke beech closed-form KL divergence likhо

D_KL(N(μ, σ²I) || N(0, I)) = (1/2) Σⱼ (μⱼ² + σ - log σⱼ² - 1)

ELBO ko "lower bound" kyun kaha jata hai?

Jensen's inequality (log E[X] ≥ E[log X]) ki wajah se, ELBO guaranteed hota hai ki ≤ log p_θ(x) hoga. ELBO maximize karna data log-likelihood par ek lower bound ko upar dhakelta hai.

KL divergence mein μ² term kya penalize karta hai?

Means jo zero (prior mean) se door hain. Encoder ko codes ko latent space ke origin ke around organize karne par majboor karta hai.

KL divergence mein -log σ² term kya encourage karta hai?

Kaafi variance rakhna encourage karta hai (bahut kam spread ki penalty). σ² term ke saath balance karta hai jo bahut zyada variance ko penalize karta hai.

Ek trained VAE se naye samples kaise generate karte hain?

(1) Prior se z ~ N(0, I) sample karo, (2) z ko decoder network p_θ(x|z) se pass karo, (3) Output distribution se x sample karo (e.g., binary images ke liye Bernoulli).

ELBO aur true posterior ke KL ke beech kya relationship hai?

log p_θ(x) = ELBO + D_KL(q_φ(z|x) || p_θ(z|x)). Kyunki KL ≥ 0, ELBO ≤ log p_θ(x). Tighter ELBO matlab better posterior approximation.

Hum typically σ² ki jagah log σ² predict kyun karte hain?

Positivity ensure karta hai (σ² = exp(log σ²) > 0 hamesha) aur numerical stability deta hai (optimization ke dauran bahut chhoti values se bachata hai).

KL annealing kya hai aur iska use kyun hota hai?

Training ke dauran KL weight ko β < 1 se β = 1 tak gradually increase karna. Posterior collapse rokne mein madad karta hai kyunki prior constraint enforce karne se pehle reconstruction seekhi ja sakti hai.

VAE objective mein, agar KL term bilkul hata do toh kya hoga?

Model standard autoencoder ban jata hai: latent space arbitrary/discontinuous ho sakta hai, codes bahut door, space mein holes, prior se sampling karke generate nahi kar sakte, overfit karne ka tendency hoti hai.

Ek VAE ki generative story kya hai?

(1) Latent code z ~ p(z) = N(0, I) sample karo, (2) Decoder network ke zariye x ~ p_θ(x|z) generate karo. Goal: