Sparse autoencoders for features

Overview

Sparse autoencoders (SAEs) are a specialized architecture designed to learn interpretable feature representations by enforcing sparsity in the hidden layer activations. Unlike standard autoencoders that compress data efficiently, SAEs prioritize learning a overcomplete basis where only a small subset of neurons activate for any given input, revealing fundamental components of the data.

The sparsity constraint fights the network's natural tendency to use all neurons weakly. Instead, we get strong, selective responses—like having experts who each speak up only about their domain.

Mathematical Foundation

Typically $m > n$ (overcomplete) where $n = \dim(\mathbf{x})$, allowing the model to learn a rich dictionary of features.

Objective Function Derivation

Step 1: Start with reconstruction loss

The base objective is mean squared error: $\mathcal{L}_{\text{recon}} = \frac{1}{N} \sum_{i=1}^N \|\mathbf{x}^{(i)} - \hat{\mathbf{x}}^{(i)}\|^2$

Why this step? We need the autoencoder to preserve information—perfect reconstruction means no information loss.

Step 2: Add sparsity constraint

The $L_0$ norm (count of non-zero elements) is ideal but non-differentiable. We approximate with:

$\mathcal{L}_{\text{sparse}} = \lambda \sum_{j=1}^m \text{KL}(\rho \| \hat{\rho}_j)$

where:

• $\rho$ is target sparsity (e.g., 0.05 means 5% average activation)
• $\hat{\rho}_j = \frac{1}{N}\sum_{i=1}^N h_j^{(i)}$ is empirical activation rate for neuron $j$
• KL divergence: $\text{KL}(\rho \| \hat{\rho}_j) = \rho \log\frac{\rho}{\hat{\rho}_j} + (1-\rho)\log\frac{1-\rho}{1-\hat{\rho}_j}$

Why KL divergence? It's a principled way to penalize deviation from target sparsity. When $\hat{\rho}_j = \rho$, KL = 0 (no penalty). When $\hat{\rho}_j$ differs, the penalty grows asymetrically—heavily punishing neurons that activate too often.

Step 3: Complete objective

The weight decay $\alpha\|\mathbf{W}\|_F^2$ prevents trivial solutions where some neurons grow arbitrarily large to compensate for others being silent.

Alternative Sparsity Penalties

L1 Regularization

Instead of KL divergence, directly penalize activation magnitudes: $\mathcal{L}_{\text{L1}} = \lambda \sum_{j=1}^m |h_j|$

Derivation insight: The $L_1$ norm is the tightest convex relaxation of $L_0$. At optimum, it encourages exactly-zero activations (unlike $L_2$ which only shrinks them).

Why this works? The $L_1$ penalty has a "corner" at zero. During gradient descent, small activations get pushed all the way to zero, creating true sparsity.

k-Sparse Constraint

Hard constraint: Keep only top-$k$ activations, zero the rest. $h_j^{\text{sparse}} = \begin{cases} h_j & \text{if } h_j \text{ in top-}k \\ 0 & \text{otherwise} \end{cases}$

Why this step? Gives exact control over sparsity level, but non-differentiable. Requires straight-through estimator: forward pass uses hard threshold, backward pass uses identity gradient.

Worked Examples

Step-by-step:

1. Input: Vectorize patch $\mathbf{x} \in \mathbb{R}^{64}$, normalize to [0,1]

   • Why? Raw pixels have arbitrary scale; normalization stabilizes training

2. Encoder: $\mathbf{h} = \sigma(W_e \mathbf{x} + \mathbf{b}_e)$ where $W_e \in \mathbb{R}^{256 \times 64}$

   • Why sigmoid? Output naturally bounded [0,1], matching interpretation as "activation probability"

3. Apply sparsity: Compute $\hat{\rho}_j = \frac{1}{N}\sum_i h_j^{(i)}$ over batch

   • Why average? Single example could be outlier; batch average is stable estimate

4. KL penalty: $\text{KL}(\rho=0.05 \| \hat{\rho}_j)$ heavily penalizes $\hat{\rho}_j > 0.05$

   • Why asymetric? Over-active neurons dominate representation; must suppress them

5. Decoder: $\hat{\mathbf{x}} = W_d \mathbf{h} + \mathbf{b}_d$ (linear, no activation)

   • Why linear? With sparse $\mathbf{h}$, decoder becomes linear combination of learned features

Result: Each of256 neurons learns edge detector (horizontal/vertical/diagonal) or Gabor-like filter at specific location/orientation. Only ~13 neurons activate per patch, each detecting present features.

Step-by-step:

1. Input: Pre-trained embedding $\mathbf{x} \in \mathbb{R}^{300}$ for word "king"

2. Encoder: $\mathbf{h} = \text{ReLU}(W_e \mathbf{x} + \mathbf{b}_e)$, $W_e \in \mathbb{R}^{1024 \times 300}$

   • Why ReLU? Natural sparsity (outputs exactly 0), interpretable as "feature presence"

3. L1 penalty: $\lambda \sum_j h_j$ directly minimizes activation sum

   • Why L1 here? Simpler than KL for high-dimensional dense embedings

4. Result: With $\rho=0.03$, ~31 neurons activate. Inspection shows:

   • Neuron 42: "royalty" (king, queen, prince → high)
   • Neuron 158: "male gender" (king, man, he → high)
   • Neuron 789: "power/authority" (king, emperor, boss → high)

Why interpretable? Overcomplete + sparse forces disentangled features. Each neuron can't "hide" multiple meanings—only one strong pattern survives the sparsity pressure.

Training Dynamics

For neuron $j$, the sparsity gradient is: $\frac{\partial \mathcal{L}_{\text{sparse}}}{\partial h_j^{(i)}} = \frac{\lambda}{N} \left( -\frac{\rho}{\hat{\rho}_j} + \frac{1-\rho}{1-\hat{\rho}_j} \right)$

Derivation:

1. $\text{KL}(\rho \| \hat{\rho}_j) = \rho \log \rho - \rho \log \hat{\rho}_j + (1-\rho)\log(1-\rho) - (1-\rho)\log(1-\hat{\rho}_j)$
2. $\frac{\partial \text{KL}}{\partial \hat{\rho}_j} = -\frac{\rho}{\hat{\rho}_j} + \frac{1-\rho}{1-\hat{\rho}_j}$
3. $\hat{\rho}_j = \frac{1}{N}\sum_i h_j^{(i)}$, so $\frac{\partial \hat{\rho}_j}{\partial h_j^{(i)}} = \frac{1}{N}$
4. Chain rule: $\frac{\partial \mathcal{L}}{\partial h_j^{(i)}} = \lambda \cdot \frac{\partial \text{KL}}{\partial \hat{\rho}_j} \cdot \frac{\partial \hat{\rho}_j}{\partial h_j^{(i)}}$

Interpretation: If $\hat{\rho}_j > \rho$, gradient is positive → reduce activation. If $\hat{\rho}_j < \rho$, gradient is negative → increase activation. The magnitude grows as $\hat{\rho}_j \to 0$ or $1$, preventing saturation.

Common Mistakes

Why it feels right: Small weights → small activations → seems like fewer neurons matter.

Why it's wrong: L2 shrinks ALL weights uniformly. You get weak activations everywhere, not selective zero/non-zero pattern. A weight of 0.01 is still non-zero.

The fix: Sparsity requires explicit $L_1$ or $L_0$-like penalties on activations, not weights. These create true zeros via thresholding or selection.

Steel-man: The intuition connects small values with "irrelevance," which is true in some contexts. But sparsity is about counts (how many non-zero), not magnitudes (how large).

Why it feels right: More constraint → more forced specialization → clearer features.

Why it's wrong: Too much sparsity starves the representation. With extreme λ, most neurons die (never activate), and the few survivors can't reconstruct the input—model converges to trivial solution.

The fix: Tune λ via validation reconstruction error. Sweet spot: maximum sparsity while maintaining reasonable reconstruction (e.g., MSE < 0.01 for normalized images).

Steel-man: More constraint does help interpretability, but only up to the information bottleneck. Beyond that, you're not learning structure—you're just failing to model the data.

Why it feels right: Standard dimensionality reduction (PCA, VAE) compresses to fewer dimensions. More seems inefficient.

Why it's wrong: Overcompleteness is the point. With $m \gg n$, each neuron can specialize to a narrow pattern. Combined with sparsity (only few active), you get a large dictionary where each entry is interpretable. Undercomplete autoencoders learn entangled, distributed codes.

The fix: Embrace overcomplete hidden layers (2-4× input dimension typical). Sparsity prevents redundancy—most features stay silent for any given input.

Applications in AI Interpretability

1. Neural Network Dissection

Train SAE on intermediate activations of a CNN (e.g., layer conv4 of ResNet). The SAE features reveal:

• Object parts (wheels, windows, faces)
• Textures (wood grain, fabric, fur)
• Abstract concepts (symmetry, curvature)

Why this works: CNN activations are dense and entangled. SAE disentangles them into human-recognizable components.

2. Language Model Probing

Apply SAE to transformer hidden states. Learned features correspond to:

• Syntactic roles (subject, object, verb)
• Semantic categories (living things, tools, emotions)
• Discourse features (question markers, negation)

Use case: Understanding what information is encoded at each layer, guiding model editing or debugging.

3. Mechanistic Interpretability

Recent work trains SAEs on GPT-2 residual stream activations. Discovered features include:

• "French language detector" (activates on French text)
• "Base64 encoder" (activates during base64 reasoning)
• "Python indentation" (activates on correctly-indented code)

Breakthrough: Shows even complex language models use modular, interpretable features internally.

Connections

• 6.3.01-Feature-visualization: SAE features can be visualized via activation maximization
• 6.3.02-Attribution-methods: SAE provides ground truth for which features caused a prediction
• 6.3.05-Concept-activation-vectors: SAE features are learned CAVs without manual annotation
• 5.2.03-Variational-autoencoders: VAEs use latent sparsity for generation; SAEs for interpretability
• 4.1.07-L1-L2-regularization: Mathematical foundation of sparsity penalties
• 3.4.05-PCA-ICA: ICA also seeks independent components, but linearly; SAE is nonlinear
• 6.3.11-Superposition-hypothesis: Explains why networks need sparsity to avoid feature interference

A sparse autoencoder learns the same way. Instead of using ALL its "brain neurons" weakly, it learns to have specialist neurons—one for "wheel detector," one for "window detector"—and only turns on the ones it needs for each picture.

The "sparse" part means "only use a few at a time." The "autoencoder" part means it learns by trying to copy its input (like looking at a LEGO car, then rebuilding it from memory). To rebuild correctly with only a few specialists, each specialist must become REALLY good at recognizing its one thing.

Why is this useful? Because then humans can look at the specialists and understand what the AI is seeing! It's like opening the AI's brain and finding neat, labeled toolbox instead of a messy pile.

Visual: Picture a SPOTLIGHT (sparsity) in a dark theater (overcomplete space), illuminating only the actors (features) relevant to the current scene (input).

#flashcards/ai-ml

What is the primary goal of a sparse autoencoder compared to a standard autoencoder? :: Learn interpretable, disentangled features by enforcing sparsity in the hidden layer, prioritizing explainability over pure compression efficiency.

Write the KL divergence sparsity penalty formula for a single neuron j :: $\text{KL}(\rho \| \hat{\rho}_j) = \rho \log\frac{\rho}{\hat{\rho}_j} + (1-\rho)\log\frac{1-\rho}{1-\hat{\rho}_j}$ where $\rho$ is target sparsity and $\hat{\rho}_j$ is empirical average activation.

Name two real-world applications where SAEs improve AI interpretability :: 1) Neural network dissection: revealing object parts and textures in CNN layers. 2) Language model probing: discovering syntactic/semantic features in transformer hidden states.

Concept Map

Hinglish (regional understanding)