Foundations — Sparse autoencoders for features

This page assumes you have seen nothing. Every letter, arrow, and squiggle that appears in the parent note is built here from the ground up, in the order you need them.

0. The picture we are always describing

Before any symbol: the whole topic is one flow of arrows.

Something goes in (a list of numbers). A first machine turns it into a wider list of numbers where most entries are zero. A second machine tries to rebuild the original from that wide-but-mostly-zero list. If it can rebuild well and stay mostly-zero, the wide list must be capturing the real "parts" the input is made of. Keep this three-box picture in your head; every symbol below labels one arrow or box in it.

1. A vector — a list of numbers with a place to live

The picture. A vector is an arrow, or equivalently a point, in a space with one axis per slot. Two numbers → a point on a flat sheet. Three numbers → a point in a room. Sixty-four numbers → a point in a space we can't draw but reason about identically.

Why the topic needs it. The input to the autoencoder — an image patch, a word embedding — is exactly such a list. In the parent note $\mathbf{x}\in {\mathbb{R}}^{64}$ means "$\mathbf{x}$ is a list of 64 numbers." We must have the concept of "list of numbers as a point" before anything else can act on it.

Reveal check:

2. Dimension $n$ and $m$ — how long the lists are

The picture. Input lives in a space of $n$ axes; the hidden representation lives in a roomier space of $m$ axes. More axes = more distinct "detectors" available.

Why the topic needs it. A normal autoencoder squeezes ($m<n$) to save space. A sparse autoencoder does the opposite: it wants a big shelf of specialist detectors and just uses a few. That "big shelf, few used" combination is the whole trick, and it needs $m>n$.

3. Weights $W$ and biases $\mathbf{b}$ — the tunable knobs

Why matrix multiply, not something fancier? Because a weighted sum is the simplest tunable way to let every input number influence every output number. Training just nudges the grid entries until useful mixtures emerge. The subscripts distinguish the two machines: $W_e,{\mathbf{b}}_e$ = encoder knobs, $W_d,{\mathbf{b}}_d$ = decoder knobs.

Reveal check:

4. The activation function $f$ / $\sigma$ / ReLU — the "should I fire?" gate

Two you must know:

Why these two, and why here? ReLU's flat-zero left half means many neurons output literal zero — that is free sparsity, no penalty needed. Sigmoid's $[0,1]$ range lets us read an output as "fraction of the time this fires," which is exactly what the KL sparsity target $\rho$ compares against (Section 8). The parent uses sigmoid for image patches and ReLU for embeddings — now you know why.

5. Encoder and decoder — the two labelled machines

• $\mathbf{h}$ = the hidden / feature list ($m$ numbers).
• $\hat{\mathbf{x}}$ (x with a hat) = the reconstruction, the decoder's best guess at the original $\mathbf{x}$. A hat always means "estimate / rebuilt version," never the true thing.

Why a decoder at all? Without it, nothing forces $\mathbf{h}$ to mean anything. The decoder is the honesty test: if features can rebuild the input, they captured its real content. This is the same idea underlying 5.2.03-Variational-autoencoders, minus the probability machinery.

Reveal check:

6. Norms — measuring "how big" and "how many"

Norms are ways to squeeze a whole vector into a single "size" number. Three matter here.

Why three? $L_0$ is the honest goal ("keep few awake") but it is a step-count — flat everywhere, so gradient descent gets no slope to follow. $L_1$ is the closest smooth-enough stand-in whose "corner at zero" still pushes small values to exactly zero (see 4.1.07-L1-L2-regularization). $L_2$ only shrinks — never zeroes — which is precisely Mistake #1 in the parent note.

7. Sums $\sum$, index $i$, batch $N$ — averaging over data

Why average over $N$? One example might be a fluke. Sparsity is a habit — "on average, is this neuron mostly quiet?" — so we measure it across a batch, not one input.

8. Sparsity target $\rho$ and rate ${\hat{\rho }}_j$ — the quota system

The picture. Imagine each neuron has a quota card: "you may light up 5% of the time." $\rho$ is the printed quota; ${\hat{\rho }}_j$ is the neuron's actual attendance record. The penalty punishes any gap.

Why KL and not just subtract? Plain $({\hat{\rho }}_j-\rho {)}^2$ would treat over- and under-firing equally. KL leans against over-firing, which matches the goal "keep most neurons quiet." That's why the parent uses it for the probability-flavoured sigmoid case.

Reveal check:

9. The greek control knobs $\lambda$ and $\alpha$

The picture. Sliders on a mixing desk. Slide $\lambda$ up → more silence pressure. But turn it too far and the representation starves (parent's Mistake #2). These are the trade-off dials of the whole loss.

10. Putting the symbols back together

Now every piece of the parent's headline loss is readable:

${\mathcal{L}}_{\text{SAE}}=\underset{\text{rebuild well (}{L}_2\text{)}}{\underbrace{\frac{1}{N}\sum _i\Vert {\mathbf{x}}^{(i)}-{\hat{\mathbf{x}}}^{(i)}{\Vert }^2}}+\underset{\text{stay quiet (quota)}}{\underbrace{\lambda \sum _j\text{KL}(\rho \Vert {\hat{\rho }}_j)}}+\underset{\text{don’t cheat (}{L}_F\text{)}}{\underbrace{\alpha \Vert W{\Vert }_F^2}}$

You can now name $\mathcal{L}$ (the total badness score to minimise), each $\sum$, each norm, $\rho$, ${\hat{\rho }}_j$, $\lambda$, $\alpha$, the hats, and the subscripts. That is the entire vocabulary of the topic.

Prerequisite map

This feeds forward into 6.3.02-Attribution-methods, 6.3.05-Concept-activation-vectors, and the 6.3.11-Superposition-hypothesis, which explains why overcomplete sparse dictionaries are needed at all.

Equipment checklist

Test yourself — you should be able to answer each before reading the main topic.