Exercises — Circuits and superposition

First, the two words in plain language:

• A circuit is a path of computation — specific neurons/heads wired together doing one algorithm (a verb: it computes).

• Superposition is a storage trick — packing more feature-directions than there are axes (a noun about capacity: it stores).

1. Superposition — "more features than dimensions via nearly-orthogonal directions" is the definition of superposition.
2. Circuit — two heads composed across layers implementing an algorithm (induction).
3. Superposition (its symptom): a polysemantic neuron is what you observe when superposition packs several feature-directions through one neuron's axis.

The dot product ${\mathbf{v}}_i\cdot {\mathbf{v}}_j$ measures alignment: $1$ = same direction, $0$ = perpendicular, $-1$ = opposite. Two features "interfere" only when their directions overlap.

Spread $\approx 1/\sqrt{n}$ shrinks as $n$ grows. So in high dimensions, random directions are almost perpendicular — huge room to pack near-orthogonal features. This is exactly why superposition scales: more axes → more "almost-perpendicular" corners to hide features in.

Every symbol (also defined in the boxes above): $m$ = number of features you can store and still decode; ${\text{SNR}}_{\min }$ = the signal ÷ noise threshold; $s$ = biggest allowed pairwise overlap ${\max }_{i\ne j}|{\mathbf{v}}_i\cdot {\mathbf{v}}_j|$; $p$ = probability a given feature is active. The $\lesssim$ means "up to a constant we set to $1$."

The bound comes straight from the boxed inequality $\frac{1}{s\sqrt{mp}}\ge {\text{SNR}}_{\min }$ (derived in full at L5.1). Rearranged it gives $m\lesssim {\text{SNR}}_{\min }^2/(p{s}^2)$. Plugging in: $m\lesssim \frac{1^2}{0.01\cdot (0.1{)}^2}=\frac{1}{0.01\cdot 0.01}=\frac{1}{0.0001}=10,000.$

So $10,000$ decodable features — vastly more than the tiny $n$ you'd need for orthogonal storage.

In Figure s01, ${\mathbf{v}}_2$ is the yellow arrow at $60^{\circ }$. An activation $\mathbf{a}=(a_1,a_2)$ literally means neuron $1$ outputs $a_1$, neuron $2$ outputs $a_2$. Since $\mathbf{a}={\mathbf{v}}_2$:

• Neuron $1$: $a_1=0.5$
• Neuron $2$: $a_2=0.866$

Notice neither neuron reads a clean "1." The information is spread across both — that spreading is superposition in action.

First check the length of ${\mathbf{v}}_2$. A projection equals a plain dot product only when the target direction is a unit vector (length $1$). Use the exact values: ${\mathbf{v}}_2=\left(\frac{1}{2},\frac{\sqrt{3}}{2}\right)$, so $\Vert {\mathbf{v}}_2{\Vert }^2={\left(\frac{1}{2}\right)}^2+{\left(\frac{\sqrt{3}}{2}\right)}^2=\frac{1}{4}+\frac{3}{4}=1\Rightarrow \Vert {\mathbf{v}}_2\Vert =1.$ Exactly unit (the $0.866$ was just the decimal for $\sqrt{3}/2$). So projection $=$ dot product here. If ${\mathbf{v}}_2$ were not unit, you would divide by its length: $\text{proj}=\frac{\mathbf{a}\cdot {\mathbf{v}}_2}{\Vert {\mathbf{v}}_2\Vert }$. Always confirm this before equating the two.

Now the readout — "how much of $\mathbf{a}$ points along ${\mathbf{v}}_2$?" — using exact values $\mathbf{a}=\left(0,\sqrt{3}\right)$: $\mathbf{a}\cdot {\mathbf{v}}_2=(0)\left(\frac{1}{2}\right)+(\sqrt{3})\left(\frac{\sqrt{3}}{2}\right)=\frac{3}{2}=\mathrm{1.5.}$ We expected $1$ (feature 2 present once). We got $1.5$. The extra $0.5$ is interference from ${\mathbf{v}}_3$ leaking through the overlap ${\mathbf{v}}_2\cdot {\mathbf{v}}_3=\left(\frac{1}{2}\right)\left(-\frac{1}{2}\right)+\left(\frac{\sqrt{3}}{2}\right)\left(\frac{\sqrt{3}}{2}\right)=-\frac{1}{4}+\frac{3}{4}=\frac{1}{2}$. Right: leak = overlap = $0.5$ (see the red dashed projection line in Figure s02). ✔

The residual stream is a shared notepad that each layer reads then writes. The previous-token head writes "the token before me was X" onto the notepad. The induction head reads that note to jump to "what followed the earlier X."

A reader can only read what's already written. So the writer (previous-token head) must run first → it must be in an earlier layer. Cross-layer composition is one-directional in time-of-computation: write-before-read forces induction depth > previous-token depth.

Geometric ceiling. The formula is $m\lesssim e^{(\text{exponent})}$ where the exponent is $c{s}^{2}n$ — an input we compute first, then feed into the exponential function $e^{(\cdot )}$. Compute the exponent: $c{s}^{2}n=(1)(0.1{)}^2(512)=(1)(0.01)(512)=\mathrm{5.12.}$ So the ceiling is $m\lesssim e^{5.12}$. Here $e\approx 2.71828$ is a fixed constant, and $e^x$ is the smooth exponential-growth function whose value at $x=5.12$ is what we want (for whole numbers $e^k$ happens to equal $e$ multiplied by itself $k$ times, but for a fractional exponent like $5.12$ it is defined by that same smooth curve, not by repeated multiplication). Numerically $e^5\approx 148.4$ and $e^{0.12}\approx 1.127$, so $e^{5.12}=e^5\cdot e^{0.12}\approx 148.4\times 1.127\approx 167.3$. Hence $m\lesssim 167$.

Decoding ceiling. ${\text{SNR}}_{\min }^2/(p{s}^2)=1^2/(0.01\cdot 0.01)=10,000$.

Which binds? The smaller ceiling is the real limit: $167<10,000$, so at this $n$ geometry binds. But geometry grows exponentially in $n$: at $n=1000$ the exponent is $c{s}^{2}n=0.01\cdot 1000=10$, giving $e^{10}\approx 22,026>10,000$, so now decoding binds. Lesson: geometry overtakes decoding as $n$ grows; in real large models ($n$ huge) the sparsity/decoding term is the binding constraint, exactly as the parent claims.

If $p\to 1$, nearly all $m$ features are on at once, so active count $k=mp\approx m$. Interference $\sim s\sqrt{k}=s\sqrt{m}$ grows with $m$. To keep noise below signal you need $s\sqrt{m}\lesssim 1\Rightarrow m\lesssim 1/s^2$ — a fixed number independent of how many directions geometry allows. The only way to store many dense features cleanly is to make them truly orthogonal → back to $m\le n$. Superposition's magic requires sparsity so that few leaks are ever summed at once.

Vectors: ${\mathbf{v}}_1=(\cos 0,\sin 0)=(1,0)$, ${\mathbf{v}}_2=(\cos 120^{\circ },\sin 120^{\circ })=(-0.5,0.866)$, ${\mathbf{v}}_3=(\cos 240^{\circ },\sin 240^{\circ })=(-0.5,-0.866)$ — the three coloured arrows of Figure s03.

Overlaps (dot products): ${\mathbf{v}}_1\cdot {\mathbf{v}}_2=(1)(-0.5)+(0)(0.866)=-0.5,$ ${\mathbf{v}}_1\cdot {\mathbf{v}}_3=-0.5,{\mathbf{v}}_2\cdot {\mathbf{v}}_3=(-0.5)(-0.5)+(0.866)(-0.866)=0.25-0.75=-\mathrm{0.5.}$ Perfectly symmetric: every overlap $=-0.5$. Note these are negative — which is why we defined $s$ with an absolute value: the tolerated overlap here is $s=|-0.5|=0.5$.

With features $1,2$ active, $\mathbf{a}={\mathbf{v}}_1+{\mathbf{v}}_2$ (the green arrow in Figure s03). Readout of feature 1: $\mathbf{a}\cdot {\mathbf{v}}_1=\underset{1}{\underbrace{{\mathbf{v}}_1\cdot {\mathbf{v}}_1}}+\underset{-0.5}{\underbrace{{\mathbf{v}}_2\cdot {\mathbf{v}}_1}}=\mathrm{0.5.}$ Expected $1$, got $0.5$ → interference $=-0.5$. The negative overlap reduces the signal (destructive leak), unlike the $60^{\circ }$ layout in Figure s01/s02 where leak was $+0.5$.

$a_j=f_3{v}_{3j}+f_{17}{v}_{17,j}+f_{42}{v}_{42,j}=(1)(0.7)+(1)(0.6)+(0)(-0.4)=\mathrm{1.3.}$ The neuron reads $1.3$ — a blend of features $3$ and $17$'s directions passing through its axis. Because three unrelated features ($3,17,42$) all have nonzero components $v_{ij}$ on this one axis, the neuron will light up for any of them. That is exactly what makes it polysemantic: one neuron, many meanings, because superposition routed several feature-directions through it. To disentangle, you'd use a tool like sparse autoencoders that re-express activations in a basis where each direction = one feature.

Setup. Reading out the target feature (projecting the activation onto its direction) gives signal $\approx 1$ plus a sum of $k=mp$ interference terms ${ϵ}_1,\dots ,{ϵ}_k$, one leak per other active feature. Each ${ϵ}_t$ is that feature's overlap with the target. We state the hypotheses explicitly, because the whole derivation rests on them:

• (H1) Zero mean. Each leak ${ϵ}_t$ has random sign, $+$ or $-$ equally likely, so its average is $\mathbb{E}[{ϵ}_t]=0$. (Feature directions are placed without a preferred orientation, so a leak is as likely to add to the readout as to subtract from it.)
• (H2) Bounded size. $|{ϵ}_t|\le s$ (the maximum allowed overlap defined in the box above), so each leak's spread satisfies $\mathbb{E}[{ϵ}_t^2]\le s^2$. Here $\mathbb{E}[{ϵ}_t^2]$ is the variance — the average of the squared leak, a measure of typical leak size that ignores sign.
• (H3) Independence. Different active features are unrelated, so their leaks ${ϵ}_t$ are (approximately) independent — knowing one tells you nothing about another.

Step 1 — WHAT: add up the leaks. Total interference is $N_{\text{tot}}={\sum }_{t=1}^k{ϵ}_t$. WHY a sum: every other active feature dumps its own overlap into the same readout, and readouts are linear (dot products), so leaks simply add.

Step 2 — WHY variances add, not magnitudes. For independent zero-mean terms, the variance of a sum is the sum of variances: $\mathrm{Var}\left({\sum }_{t=1}^k{ϵ}_t\right)={\sum }_{t=1}^k\mathrm{Var}({ϵ}_t)\le k{s}^2.$ The cross-terms $\mathbb{E}[{ϵ}_t{ϵ}_u]$ ($t\ne u$) vanish precisely because of (H1)+(H3): independent, mean-zero variables have zero covariance ($\mathbb{E}[{ϵ}_t{ϵ}_u]=\mathbb{E}[{ϵ}_t]\mathbb{E}[{ϵ}_u]=0$). So the typical size (standard deviation = square-root of variance) of the total interference is $N=\sqrt{\mathrm{Var}}\lesssim s\sqrt{k}=s\sqrt{mp}.$ WHY $\sqrt{k}$, not $k$: this is the drunkard's-walk law. If we had (wrongly) added magnitudes we'd get $ks$; but signed independent steps cancel on average, so the net displacement grows only like $\sqrt{\text{number of steps}}$. That square-root is the whole reason superposition is powerful — noise grows slowly.

Step 3 — WHAT the picture looks like. Signal is a fixed arrow of length $\approx 1$ along the target direction; the $k$ leaks are tiny random $\pm$ nudges that pile up to a fuzzy cloud of radius $\approx s\sqrt{mp}$ around the signal's tip. Decoding succeeds when that cloud stays smaller than the signal.

Step 4 — impose decodability. Using the SNR convention fixed at the top (signal ÷ noise $\ge {\text{SNR}}_{\min }$), with signal $=1$ and noise $N=s\sqrt{mp}$: $\frac{\text{signal}}{\text{noise}}=\frac{1}{s\sqrt{mp}}\ge {\text{SNR}}_{\min }.$ Multiply both sides by $s\sqrt{mp}$ (positive) and divide by ${\text{SNR}}_{\min }$ (positive): $s\sqrt{mp}\le \frac{1}{{\text{SNR}}_{\min }}.$ Square both sides (both are positive, so the inequality direction is preserved): $s^{2}mp\le \frac{1}{{\text{SNR}}_{\min }^2}⟹\boxed{m\lesssim \frac{1}{{\text{SNR}}_{\min }^{2}p{s}^2}}.$ Where the emergence comes from: the SNR and the overlap $s$ appear squared because we squared the $s\sqrt{mp}$ random-walk relation — the $\sqrt{k}$ noise law forces exactly one squaring. Writing the tolerated-noise budget $1/{\text{SNR}}_{\min }$ as ${\text{SNR}}_{\min }$ recovers the parent's boxed form $m\lesssim {\text{SNR}}_{\min }^2/(p{s}^2)$; with ${\text{SNR}}_{\min }=1$ both forms coincide and give the $10,000$ of L2.1. ∎

Ablation = force the heads' outputs to zero and re-run. If behavior collapses, the heads cause it.

Causal effect $=\frac{\text{baseline}-\text{ablated}}{\text{baseline}}=\frac{0.92-0.31}{0.92}=\frac{0.61}{0.92}\approx \mathrm{0.663.}$

So ≈66.3% of the behavior vanishes when the circuit is removed. A reasonable rule: call it a genuine causal circuit if the drop is large and specific — e.g. effect $>0.5$ on the target task while random-head ablations of equal size cause $<0.1$. Here $0.663>0.5$: strong causal evidence, matching the parent's claim that ablating induction heads sharply drops induction accuracy. This same ablation logic underlies attention-head interpretability and concept activation vectors.

Superposition stores more features than dimensions by using nearly-orthogonal directions. A neuron responding to many unrelated features is called polysemantic. Interference from $k$ active features grows like ==$s\sqrt{k}$ (random-walk). Induction heads must be deeper== than the previous-token heads they read from. The decodable feature ceiling is ::: $m\lesssim {\text{SNR}}_{\min }^2/(p{s}^2)$.