Autoencoders fundamentals — Hinglish notes

4.5.2 · AI-ML › Generative Models

What is an Autoencoder?

WHY teen components? Encoder discover karta hai kya important hai, bottleneck compression force karta hai (network ko sirf memorize karne se rokta hai), aur decoder test karta hai ki compressed representation kaafi hai ya nahi.

WHAT "auto" matlab hai? Network khud se seekhta hai — target output wohi hai jo input hai. Labeled data ki koi zaroorat nahi.

Architecture Deep-Dive

Basic Structure

Sabse simple autoencoder symmetric hota hai:

Encoder: $x \in R^{n} W_{1}, b_{1} h_{1} σ h_{2} W_{2}, b_{2} z \in R^{m}$

Decoder (mirror): $z W_{3}, b_{3} h_{3} σ h_{4} W_{4}, b_{4} \hat{x} \in R^{n}$

jahan $m < n$ (bottleneck), $σ$ activation hai (ReLU, tanh), $W_{i}$ weight matrices hain.

WHY symmetric? Zaroori nahi, lekin common hai. Yeh compression/decompression ki symmetry mirror karta hai aur hyperparameters kam karta hai.

Deriving the Loss Function

Goal: Reconstruction $\hat{x}$ ko original $x$ ke kareeb laao.

Continuous data ke liye (images, sensor readings), Gaussian noise assume karo: $p (x ∣ \hat{x}) = N (x ∣ \hat{x}, σ^{2} I)$

Maximum likelihood estimation deta hai: $max p (x ∣ \hat{x}) = max exp (- \frac{1}{2 σ ^{2}} ∥ x - \hat{x} ∥^{2})$

Negative log lo (minimization ke liye):

Binary data ke liye (black/white images, binary vectors), binary cross-entropy use karo:

WHY alag losses? Loss data type aur assumed noise model se match karta hai. Continuous ke liye MSE, binary ke liye BCE.

Training Process

Forward Pass

Encode: $z = f_{ϕ} (x) = σ (W_{2} σ (W_{1} x + b_{1}) + b_{2})$
Decode: $\hat{x} = g_{θ} (z) = σ (W_{4} σ (W_{3} z + b_{3}) + b_{4})$
Loss compute karo: $L = ∥ x - \hat{x} ∥^{2}$

Backward Pass (Gradient Derivation)

Chain rule decoder → bottleneck → encoder ke through:

$\frac{\partial L}{\partial W _{1}} = \frac{\partial L}{\partial x ^} \frac{\partial x ^}{\partial z} \frac{\partial z}{\partial W _{1}}$

WHY yeh important hai? Gradient bottleneck ke through flow karta hai. Agar $z$ bahut chhota ho, gradients vanish ho jaate hain. Agar bahut bada ho, network sirf memorize karta hai.

HOW memorization rokein?

$dim (z)$ ko itna chhota rakho ki compression force ho
Regularization add karo (baad mein: VAE KL divergence add karta hai, sparse autoencoders sparsity penalty add karte hain)

Worked example Worked Example 1: Linear Autoencoder

Problem: 2D points $(x_{1}, x_{2})$ jo line $x_{2} = 2 x_{1}$ pe hain unhe 1D mein compress karna seekho. Setup:

Input: $x = [x_{1}, x_{2}]^{T} \in R^{2}$
Latent: $z \in R^{1}$ (bottleneck = 1)
Koi activation nahi (linear autoencoder) Architecture:
Encoder: $z = w_{1} x_{1} + w_{2} x_{2}$
Decoder: $\overset{x}{^}_{1} = u_{1} z, \overset{x}{^}_{2} = u_{2} z$ Loss: $L = (x_{1} - u_{1} z)^{2} + (x_{2} - u_{2} z)^{2}$

Substitute $z = w_{1} x_{1} + w_{2} x_{2}$ : $L = (x_{1} - u_{1} (w_{1} x_{1} + w_{2} x_{2}))^{2} + (x_{2} - u_{2} (w_{1} x_{1} + w_{2} x_{2}))^{2}$

Optimum pe ( $x_{2} = 2 x_{1}$ data ke liye): Network seekhta hai $w_{2} \approx 2 w_{1}$ aur $u_{2} \approx 2 u_{1}$ .

WHY? Single number $z$ ko ek hi degree of freedom capture karna hota hai (line pe position). 2:1 ratio constraint encode karta hai.

Interpretation: Linear autoencoder PCA (Principal Component Analysis) karta hai.

Worked example Worked Example 2: MNIST Digit Compression

Problem: 28×28 = 784-pixel grayscale digit images ko 32 dimensions mein compress karo.

Architecture:

Input: $x \in R^{784}$ (flattened image)
Encoder: 784 → 128 (ReLU) → 32 (linear)
Decoder: 32 → 128 (ReLU) → 784 (sigmoid)
Loss: BCE (pixels 0-1 normalized hain)

Training:

z = encoder(x)          # shape: (batch, 32)
x_hat = decoder(z)      # shape: (batch, 784)
loss = BCE(x, x_hat)

WHY sigmoid output? BCE ke liye output [0,1] mein chahiye (pixel intensities).

WHY 32 dimensions? Empirically: 784 → 32 matlab ~25× compression. Chhota (jaise 8) detail lose karta hai, bada (jaise 128) zyaada compress nahi karta.

Result: Training ke baad, $z$ digit identity capture karta hai. Similar digits (jaise alag-alag "3"s) latent space mein cluster karte hain.

The Bottleneck: Why Compression Matters

Latent dimension $d_{z}$ behavior decide karta hai:

$d_{z}$	Behavior	Problem
$d_{z} \geq d_{x}$	Identity mapping	Koi compression nahi, memorization
$d_{z} \approx d_{x} /2$	Moderate compression	Balanced
$d_{z} ≪ d_{x}$	Strong compression	Important details lose ho sakti hain

Capacity ka derivation:

Information content: $I (x) \leq d_{x} lo g_{2} (L)$ bits ( $L$ quantization levels per dimension ke liye).

Bottleneck capacity: $I (z) \leq d_{z} lo g_{2} (L_{z})$ bits.

Perfect reconstruction ke liye: $d_{z} lo g_{2} (L_{z}) \geq I (x)$ (Shannon's source coding theorem).

WHY compress kar sakte hain? Real data ki intrinsic dimensionality $d_{intrinsic} ≪ d_{x}$ hoti hai. Faces ki images ek low-dimensional manifold (face space) pe hoti hain, $R^{784}$ ko uniformly fill nahi karti.

Common mistake Common Mistake: Bottleneck Ko Bahut Wide Banana

Galat intuition: "Bada latent space → better reconstruction."

Kyun sahi lagta hai: Zyaada capacity matlab network zyaada information store kar sakta hai.

Problem: Agar $d_{z} \geq d_{x}$ ho, network seekh sakta hai: $f_{ϕ} (x) = x, g_{θ} (z) = z$

Yeh identity mapping hai — koi compression nahi, koi learned structure nahi. Latent space meaningless hai.

Fix:

$d_{z} < d_{x}$ set karo (typically $d_{x}$ ka 10-50%)
Regularization add karo (sparsity, noise, VAEs mein KL divergence)
Monitor karo: Reconstruction error vs. $d_{z}$ plot karo. "Elbow" dhundo jahan error improve karna band kar de.

Steel-man: "Zyaada capacity help karta hai" ki intuition overall network ke liye sahi hai, lekin bottleneck constrained hona chahiye taaki learning force ho.

Common mistake Common Mistake: Data Type Ke Liye Galat Loss Use Karna

Galat code:

# Binary image (0/1 pixels), but using MSE
loss = nn.MSELoss()

Kyun sahi lagta hai: MSE "default" loss hai, aur yeh distance measure karta hai.

Problem: MSE pixel errors ko linearly treat karta hai. Binary data ke liye, $p (x ∣0.1)$ bahut alag hona chahiye $p (x ∣0.9)$ se jab $x = 1$ ho, lekin MSE similar penalty deta hai.

BCE sahi se Bernoulli likelihood model karta hai: $- lo g (0.9) = 0.11$ (good), $- lo g (0.1) = 2.30$ (bad). Fix:

# Binary data → BCE
loss = nn.BCELoss()
# Continuous data (photos, sensor) → MSE
loss = nn.MSELoss()

Properties of Latent Space

Interpolation

Kya hum do images ke beech smoothly morph kar sakte hain?

$x_{A}, x_{B}$ diye hue, latent codes compute karo: $z_{A} = f_{ϕ} (x_{A}), z_{B} = f_{ϕ} (x_{B})$

Interpolate karo: $z_{t} = (1 - t) z_{A} + t z_{B}, t \in [0, 1]$

Decode karo: $x_{t} = g_{θ} (z_{t})$

WHAT dikhega? Smooth transitions (jaise "3" dheere dheere "8" mein morph ho jaata hai).

WHY? Encoder similar inputs ko nearby latent points pe map karta hai. Latent space mein linear interpolation → data space mein semantic interpolation.

Limitation: Basic autoencoders mein latent space mein gaps hote hain. Har $z$ realistic images decode nahi karta. (VAEs isko fix karte hain continuous distributions force karke.)

Dimensionality Reduction Comparison

Method	Type	Latent Space Nonlinear?
PCA	Linear	Orthogonal
Autoencoder (linear)	Linear	PCA ke equivalent
Autoencoder (nonlinear)	Nonlinear	Learned manifold
t-SNE	Nonlinear	Visualization only

Proof ki linear AE = PCA:

Minimize karo $L = ∥ x - W^{T} x ∥^{2}$ jahan $W \in R^{n \times k}$ (encoder), $W^{T}$ (decoder).

Gradient lo: $\frac{\partial L}{\partial W} = 0$ deta hai covariance matrix $E [x x^{T}]$ ke eigenvectors.

Yahi PCA hai. WHY tab autoencoders use karein? Nonlinear activations unhe curved manifolds seekhne deti hain, sirf hyperplanes nahi.

Recall Ek 12-Saal ke Bacche Ko Samjhao

Socho tum apne dost ko photos bhej rahe ho, lekin internet slow hai. Tum puri 1MB photo nahi bhej sakte. Toh tum likhte ho: "Yeh ek cat hai, orange, couch pe leti hai, sunlight left se aa rahi hai." Yeh shayad 50 bytes hogi! Tera dost yeh description padh ke ek picture banaata hai. Perfect nahi hogi, lekin main idea capture karega.

Yahi autoencoder hai! Encoder (tum) photo dekhta hai aur key facts likh deta hai. Decoder (dost) woh facts padhta hai aur photo recreate karta hai. Latent code woh chhoti description hai.

Training practice jaisi hai: Tum photo bhejte ho, dost draw karta hai, tum compare karte ho. Agar cat kutta lagti hai, tum apni description adjust karte ho ("add: pointy ears, whiskers"). Kai tries ke baad, descriptions better hoti jaati hain, aur drawings originals jaisi dikhne lagti hain.

Jadoo yeh hai: description photo ke comparison mein tiny hai, lekin sab important cheezein contain karti hai!

Use Cases

Dimensionality reduction: High-D data visualize karo (PCA, t-SNE ka alternative)
Anomaly detection: High reconstruction error → outlier (jaise fraud detection)
Denoising: Noisy inputs pe train karo, clean targets ke saath (denoising autoencoders)
Pretraining: Supervised task ke liye encoder ko initialization ke roop mein use karo (transfer learning)
Data compression: Lossy compression (lekin JPEG/H.264 se competitive nahi)

WHY anomaly detection? Network "normal" data distribution reconstruct karna seekhta hai. Unusual inputs (anomalies) poorly reconstruct hote hain kyunki woh learned patterns mein fit nahi hote, jo unhe detectable banata hai.

Connections

Principal Component Analysis (PCA): Linear autoencoders equivalent hote hain
Variational Autoencoders (VAE): Generation ke liye probabilistic latent space add karte hain
Denoising Autoencoders: Robustness ke liye corrupted inputs pe train karo
Sparse Autoencoders: Latent activations pe sparsity penalty add karo
Encoder-Decoder Architectures: Same structure, seq2seq models mein use hota hai
Latent Space Representations: Generative models mein central concept
Dimensionality Reduction Techniques: Autoencoders ek approach hain
Unsupervised Learning: Autoencoders ko koi labels nahi chahiye

#flashcards/ai-ml

Autoencoder ke teen components kya hote hain? :: Encoder (input ko latent code mein map karta hai), Latent space (compressed representation), Decoder (latent code se input reconstruct karta hai)

Autoencoder ka training objective kya hota hai?

Input x aur reconstructed output x̂ ke beech reconstruction loss minimize karna, typically MSE ya BCE use karke depending on data type

Autoencoders mein bottleneck dimension kyun important hai?

Yeh input dimension se chhota hona chahiye taaki compression force ho aur memorization na ho. Agar bahut bada ho, network identity mapping seekhta hai; agar bahut chhota ho, important information lose ho jaati hai.

Binary data ke liye autoencoders mein kaun sa loss function use karna chahiye?

Binary Cross-Entropy (BCE), Bernoulli likelihood se derive kiya: -Σ[x_i log(x̂_i) + (1-x_i)log(1-x̂_i)]

Continuous data ke liye autoencoders mein kaun sa loss function use karna chahiye?

Mean Squared Error (MSE): (1/n)Σ(x_i - x̂_i)², Gaussian likelihood assumption se derive kiya

Linear autoencoders aur PCA ka kya relationship hai?

Linear autoencoders mathematically PCA ke equivalent hote hain — yeh data covariance matrix ke eigenvectors seekhte hain jo maximum variance capture karte hain

Autoencoder latent space mein interpolation kaise kaam karta hai?

Do inputs ko z_A aur z_B mein encode karo, linearly interpolate karo z_t = (1-t)z_A + t z_B, phir z_t ko decode karo taaki original inputs ke beech smooth transitions milein

Data compression ke liye autoencoders kyun kaam karte hain?

Real data ki intrinsic dimensionality ambient dimensionality se bahut kam hoti hai redundancy aur structure ki wajah se. Autoencoders yeh low-dimensional manifold discover karte hain.

Bottleneck size set karte waqt ek common mistake kya hai?

Usse bahut bada banana (d_z ≥ d_x) jo identity mapping allow karta hai bina useful compressed representations seekhe. Fix: d_z ko d_x se significantly chhota set karo

Anomaly detection ke liye autoencoders kaise use hote hain?

Network "normal" data ko ache se reconstruct karna seekhta hai. Anomalies mein high reconstruction error hoti hai kyunki woh learned distribution mein fit nahi hote, jo unhe detectable banata hai.

Image reconstruction ke liye decoder output layer mein typically kaun sa activation function use hota hai?

Binary/normalized images ke liye Sigmoid (outputs [0,1] mein), ya data range ke according linear/tanh. Output distribution se match karna zaroori hai.

Ek autoencoder ko identity function seekhne se kya rokta hai?

Dimensionality bottleneck constraint (d_z < d_x) aur/ya regularization techniques (sparsity penalties, noise injection, VAEs mein KL divergence)