Autoencoders fundamentals

What is an Autoencoder?

WHY three components? The encoder discovers what's important, the bottleneck forces compression (preventing the network from just memorizing), and the decoder tests if the compressed representation is sufficient.

WHAT makes it "auto"? The network teaches itself – the target output is the input. No labeled data required.

Architecture Deep-Dive

Basic Structure

The simplest autoencoder is symmetric:

Encoder: $\mathbf{x} \in \mathbb{R}^n \xrightarrow{W_1, b_1} \mathbf{h}_1 \xrightarrow{\sigma} \mathbf{h}_2 \xrightarrow{W_2, b_2} \mathbf{z} \in \mathbb{R}^m$

Decoder (mirror): $\mathbf{z} \xrightarrow{W_3, b_3} \mathbf{h}_3 \xrightarrow{\sigma} \mathbf{h}_4 \xrightarrow{W_4, b_4} \hat{\mathbf{x}} \in \mathbb{R}^n$

where $m< n$ (bottleneck), $\sigma$ is activation (ReLU, tanh), $W_i$ are weight matrices.

WHY symmetric? Not required, but common. It mirrors the compression/decompression symmetry and reduces hyperparameters.

Deriving the Loss Function

Goal: Make reconstruction $\hat{\mathbf{x}}$ close to original $\mathbf{x}$ .

For continuous data (images, sensor readings), assume Gaussian noise: $p(\mathbf{x}|\hat{\mathbf{x}}) = \mathcal{N}(\mathbf{x}|\hat{\mathbf{x}}, \sigma^2 I)$

Maximum likelihood estimation gives: $\max p(\mathbf{x}|\hat{\mathbf{x}}) = \max \exp\left(-\frac{1}{2\sigma^2}\|\mathbf{x} - \hat{\mathbf{x}}\|^2\right)$

Taking negative log (for minimization):

For binary data (black/white images, binary vectors), use binary cross-entropy:

WHY different losses? The loss matches the data type and assumed noise model. MSE for continuous, BCE for binary.

Training Process

Forward Pass

Encode: $\mathbf{z} = f_\phi(\mathbf{x}) = \sigma(W_2 \sigma(W_1 \mathbf{x} + b_1) + b_2)$
Decode: $\hat{\mathbf{x}} = g_\theta(\mathbf{z}) = \sigma(W_4 \sigma(W_3 \mathbf{z} + b_3) + b_4)$
Compute loss: $\mathcal{L} = \|\mathbf{x} - \hat{\mathbf{x}}\|^2$

Backward Pass (Gradient Derivation)

Chain rule through decoder → bottleneck → encoder:

$\frac{\partial \mathcal{L}}{\partial W_1} = \frac{\partial \mathcal{L}}{\partial \hat{\mathbf{x}}} \frac{\partial \hat{\mathbf{x}}}{\partial \mathbf{z}} \frac{\partial \mathbf{z}}{\partial W_1}$

WHY this matters? The gradient flows through the bottleneck. If $\mathbf{z}$ is too small, gradients vanish. If too large, the network just memorizes.

HOW to prevent memorization?

Make $\dim(\mathbf{z})$ small enough to force compression
Add regularization (later: VAE adds KL divergence, sparse autoencoders add sparsity penalty)

Worked example Worked Example 1: Linear Autoencoder

Problem: Learn to compress 2D points $(x_1, x_2)$ lying on line $x_2 = 2x_1$ to1D. Setup:

Input: $\mathbf{x} = [x_1, x_2]^T \in \mathbb{R}^2$
Latent: $z \in \mathbb{R}^1$ (bottleneck = 1)
No activation (linear autoencoder) Architecture:
Encoder: $z = w_1 x_1 + w_2 x_2$
Decoder: $\hat{x}_1 = u_1 z, \quad \hat{x}_2 = u_2 z$ Loss: $\mathcal{L} = (x_1 - u_1 z)^2 + (x_2 - u_2 z)^2$

Substitute $z = w_1 x_1 + w_2 x_2$ : $\mathcal{L} = (x_1 - u_1(w_1 x_1 + w_2 x_2))^2 + (x_2 - u_2(w_1 x_1 + w_2 x_2))^2$

At optimum (for data on $x_2 = 2x_1$ ): Network learns $w_2 \approx 2w_1$ and $u_2 \approx 2u_1$ .

WHY? The single number $z$ must capture the only degree of freedom (position along the line). The 2:1 ratio encodes the constraint.

Interpretation: Linear autoencoder performs PCA (Principal Component Analysis).

Worked example Worked Example 2: MNIST Digit Compression

Problem: Compress 28×28 = 784-pixel grayscale digit images to 32dimensions.

Architecture:

Input: $\mathbf{x} \in \mathbb{R}^{784}$ (flattened image)
Encoder: 784 → 128 (ReLU) → 32 (linear)
Decoder: 32 → 128 (ReLU) → 784 (sigmoid)
Loss: BCE (pixels are 0-1normalized)

Training:

z = encoder(x)          # shape: (batch, 32)
x_hat = decoder(z)      # shape: (batch, 784)
loss = BCE(x, x_hat)

WHY sigmoid output? BCE requires output in [0,1] (pixel intensities).

WHY 32 dimensions? Empirical: 784 → 32 is ~25× compression. Smaller (e.g., 8) loses detail, larger (e.g., 128) doesn't compress much.

Result: After training, $\mathbf{z}$ captures digit identity. Similar digits (e.g., different "3"s) cluster in latent space.

The Bottleneck: Why Compression Matters

The latent dimension $d_z$ determines behavior:

$d_z$	Behavior	Problem
$d_z \geq d_x$	Identity mapping	No compression, memorization
$d_z \approx d_x / 2$	Moderate compression	Balanced
$d_z \ll d_x$	Strong compression	May lose important details

Derivation of capacity:

Information content: $I(\mathbf{x}) \leq d_x \log_2(L)$ bits (for $L$ quantization levels per dimension).

Bottleneck capacity: $I(\mathbf{z}) \leq d_z \log_2(L_z)$ bits.

For perfect reconstruction: $d_z \log_2(L_z) \geq I(\mathbf{x})$ (Shannon's source coding theorem).

WHY can we compress? Real data has intrinsic dimensionality $d_{\text{intrinsic}} \ll d_x$ . Images of faces live on a low-dimensional manifold (face space), not uniformly filling $\mathbb{R}^{784}$ .

Common mistake Common Mistake: Making the Bottleneck Too Wide

Wrong intuition: "Bigger latent space → better reconstruction."

Why it feels right: More capacity means network can store more information.

The problem: If $d_z \geq d_x$ , network can learn: $f_\phi(\mathbf{x}) = \mathbf{x}, \quad g_\theta(\mathbf{z}) = \mathbf{z}$

This is identity mapping – no compression, no learned structure. The latent space is meaningless.

The fix:

Set $d_z < d_x$ (typically 10-50% of $d_x$ )
Add regularization (sparsity, noise, KL divergence in VAEs)
Monitor: Plot reconstruction error vs. $d_z$ . Find "elbow" where error stops improving.

Steel-man: The intuition that "more capacity helps" is correct for the overall network, but the bottleneck must be constrained to force learning.

Common mistake Common Mistake: Using Wrong Loss for Data Type

Wrong code:

# Binary image (0/1 pixels), but using MSE
loss = nn.MSELoss()

Why it feels right: MSE is the "default" loss, and it measures distance.

The problem: MSE treats pixel errors linearly. For binary data, $p(x|0.1)$ should be very different from $p(x|0.9)$ when $x=1$ , but MSE gives similar penalty.

BCE correctly models the Bernoulli likelihood: $-\log(0.9) = 0.11$ (good), $-\log(0.1) = 2.30$ (bad). The fix:

# Binary data → BCE
loss = nn.BCELoss()
# Continuous data (photos, sensor) → MSE
loss = nn.MSELoss()

Properties of Latent Space

Interpolation

Can we smoothly morph between two images?

Given $\mathbf{x}_A, \mathbf{x}_B$ , compute latent codes: $\mathbf{z}_A = f_\phi(\mathbf{x}_A), \quad \mathbf{z}_B = f_\phi(\mathbf{x}_B)$

Interpolate: $\mathbf{z}_t = (1-t)\mathbf{z}_A + t\mathbf{z}_B, \quad t \in [0,1]$

Decode: $\mathbf{x}_t = g_\theta(\mathbf{z}_t)$

WHAT do we see? Smooth transitions (e.g., "3" gradually morphs into "8").

WHY? The encoder maps similar inputs to nearby latent points. Linear interpolation in latent space → semantic interpolation in data space.

Limitation: Basic autoencoders have gaps in latent space. Not all $\mathbf{z}$ decode to realistic images. (VAEs fix this by forcing continuous distributions.)

Dimensionality Reduction Comparison

Method	Type	Latent Space Nonlinear?
PCA	Linear	Orthogonal
Autoencoder (linear)	Linear	Equivalent to PCA
Autoencoder (nonlinear)	Nonlinear	Learned manifold
t-SNE	Nonlinear	Visualization only

Proof that linear AE = PCA:

Minimize $\mathcal{L} = \|\mathbf{x} - W^T\mathbf{x}\|^2$ where $W \in \mathbb{R}^{n \times k}$ (encoder), $W^T$ (decoder).

Taking gradient: $\frac{\partial \mathcal{L}}{\partial W} = 0$ gives eigenvectors of covariance matrix $\mathbb{E}[\mathbf{x}\mathbf{x}^T]$ .

This is PCA. WHY use autoencoders then? Nonlinear activations let them learn curved manifolds, not just hyperplanes.

Recall Explain to a 12-Year-Old

Imagine you're sending photos to a friend, but the internet is slow. You can't send the full 1MB photo. So you write down: "It's a cat, orange, lying on a couch, sunlight from left." That's maybe 50bytes! Your friend reads this description and draws a picture. It won't be perfect, but it'll capture the main idea.

That's an autoencoder! The encoder (you) looks at the photo and writes down the key facts. The decoder (friend) reads those facts and recreates the photo. The latent code is the short description.

Training is like practicing: You send the photo, friend draws it, you compare. If the cat looks like a dog, you adjust your description ("add: pointy ears, whiskers"). After many tries, the descriptions get better, and the drawings look more like the originals.

The magic: the description is tiny compared to the photo, but contains everything important!

Use Cases

Dimensionality reduction: Visualize high-D data (alternative to PCA, t-SNE)
Anomaly detection: High reconstruction error → outlier (e.g., fraud detection)
Denoising: Train on noisy inputs, clean targets (denoising autoencoders)
Pretraining: Encoder as initialization for supervised task (transfer learning)
Data compression: Losy compression (though not competitive with JPEG/H.264)

WHY anomaly detection? Network learns "normal" data distribution. Unusual inputs (anomalies) reconstruct poorly because they don't fit learned patterns.

Connections

Principal Component Analysis (PCA): Linear autoencoders are equivalent
Variational Autoencoders (VAE): Add probabilistic latent space for generation
Denoising Autoencoders: Train with corrupted inputs for robustness
Sparse Autoencoders: Add sparsity penalty on latent activations
Encoder-Decoder Architectures: Same structure, used in seq2seq models
Latent Space Representations: Central concept in generative models
Dimensionality Reduction Techniques: Autoencoders are one approach
Unsupervised Learning: Autoencoders require no labels

#flashcards/ai-ml

What are the three components of an autoencoder? :: Encoder (maps input to latent code), Latent space (compressed representation), Decoder (reconstructs input from latent code)

What is the training objective of an autoencoder?

Minimize reconstruction loss between input x and reconstructed output x̂, typically using MSE or BCE depending on data type

Why is the bottleneck dimension important in autoencoders?

It must be smaller than input dimension to force compression and prevent memorization. If too large, network learns identity mapping; if too small, loses important information.

What loss function should be used for binary data in autoencoders?

Binary Cross-Entropy (BCE), derived from Bernoulli likelihood: -Σ[x_i log(x̂_i) + (1-x_i)log(1-x̂_i)]

What loss function should be used for continuous data in autoencoders?

Mean Squared Error (MSE): (1/n)Σ(x_i - x̂_i)², derived from Gaussian likelihood assumption

What is the relationship between linear autoencoders and PCA?

Linear autoencoders are mathematically equivalent to PCA—they learn eigenvectors of the data covariance matrix that capture maximum variance

How does interpolation work in autoencoder latent space?

Encode two inputs to z_A and z_B, interpolate linearly z_t = (1-t)z_A + t z_B, then decode z_t to get smooth transitions between original inputs

Why do autoencoders work for data compression?

Real data has intrinsic dimensionality much lower than ambient dimensionality due to redundancy and structure. Autoencoders discover this low-dimensional manifold.

What is a common mistake when setting bottleneck size?

Making it too large (d_z ≥ d_x) allows identity mapping without learning useful compressed representations. Fix: set d_z significantly smaller than d_x

How are autoencoders used for anomaly detection?

Network learns to reconstruct "normal" data well. Anomalies have high reconstruction error because they don't fit the learned distribution, making them detectable.

What activation function is typically used in the decoder output layer for image reconstruction?

Sigmoid for binary/normalized images (outputs in [0,1]), or linear/tanh depending on data range. Must match the expected output distribution.

What prevents an autoencoder from learning the identity function?

The dimensionality bottleneck constraint (d_z < d_x) and/or regularization techniques (sparsity penalties, noise injection, KL divergence in VAEs)

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Autoencoder ek neural network hai jo data ko compress karke chota representation banata hai (encoding), phir wapas original form mein reconstruct karta hai (decoding). Sochiye jaise ap ek bada suitcase pack kar rahe ho – apko decide karna padta hai ki kya zaroori hai, kya redundant hai, aur destination pe jake sab kaise wapas unpack karenge. Real-world data mein bohot redundancy hoti hai – images mein spatial patterns, text mein semantic structure. Autoencoder ek information bottleneck create karta hai, jisse network ko underlying structure discover karna force hota hai.

Teen components hote hain: Encoder input ko latent space mein map karta hai (compressed form), Latent space woh small representation hai (jaise 784 pixels ko 32 numbers mein compress karna), aur Decoder us compressed code se original input ko reconstruct karta hai. Training objective simple hai: reconstruction loss minimize karo – matlab input aur reconstructed output ka difference kam karo, using MSE (continuous data ke liye) ya BCE (binary data ke liye).

Autoencoder ka magic bottleneck dimension mein hai. Agar latent space bohot bada hai toh network sirf memorize kar lega (identity mapping), kuch seekhega nahi. Agar bohot chota hai toh important details kho jayengi. Sahi size se network ko data ka intrinsic structure seekhna padta hai – jaise MNIST digits mein 784 pixels hain but actually wo 10-20 dimensions ki manifold pe live karte hain (digit identity, rotation, thickness). Latent space mein similar inputs (jaise different "3"s) pas-paas cluster hote hain, aur interpolation se smooth transitions milte hain (jaise "3" gradually "8" mein morph ho). Ye unsupervised learning hai – koi labels nahi chahiye, network khud seekhta hai ki data mein kya important hai!

What is an Autoencoder?

Architecture Deep-Dive

Basic Structure

Deriving the Loss Function

Training Process

Forward Pass

Backward Pass (Gradient Derivation)

The Bottleneck: Why Compression Matters

Properties of Latent Space

Interpolation

Dimensionality Reduction Comparison

Use Cases

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Generative Models

Connections