Circuits and superposition

What Are Circuits?

Why Circuits Form: From First Principles

Start with gradient descent on a neural network. The loss function $\mathcal{L}$ depends on all parameters $\theta$ , so:

$\frac{\partial \mathcal{L}}{\partial \theta_i} = \sum_{\text{paths}} \frac{\partial \mathcal{L}}{\partial \text{output}} \cdot \frac{\partial \text{output}}{\partial \theta_i}$

Key insight: Not all paths contribute equally. Through gradient routing, only certain weight combinations get reinforced for a given task. Over training:

Specialization: Weights form strong connections along task-relevant paths
Pruning: Irrelevant paths have near-zero gradients, their weights stay small
Emergence: The high-weight paths form interpretable circuits

Worked example Example: Induction Circuit in a Transformer

Task: Complete "When Mary and John went to the store, Mary gave a drink to..." → "John" Circuit components (a two-head composition across layers):

Previous-token head (an early layer, e.g. layer 2 in a small GPT-2): Attends to the token immediately before the current position → writes "the token before me was X" into the residual stream
Induction head (a later layer — empirically GPT-2 induction heads sit in middle/deep layers such as layers 5, 7, 9): Attends to tokens after previous occurrences of the current token → when processing the second "Mary", it uses the previous-token head's output to look at what came after the first "Mary"
Copy mechanism: The induction head's OV circuit moves the attended token's information toward the output logits

Why this works:

Step 1 (early layer): The previous-token head tags each position with "what preceded me"
Step 2 (later layer): At the second "Mary", the induction head searches history for the pattern [X] → Mary and lands on the position right after the first "Mary" (i.e. "and John")
Step 3: Attends to "John", copies it toward the prediction Why the two heads live in different layers: The induction head reads the output that the previous-token head wrote — this cross-layer composition only works if the writer runs first, so induction heads must be deeper than the previous-token heads they depend on. Verification: Ablate (zero out) the identified induction heads → induction accuracy drops sharply. The circuit is causal, not merely correlated.

What Is Superposition?

Deriving Superposition Capacity from Geometry

Setup: Model has $n$ dimensions, must represent $m$ features ( $m > n$ ). Each feature $i$ appears with probability $p$ (sparsity).

Claim: If features are sparse, superposition lets $m$ vastly exceed $n$ .

Derivation:

Orthogonal baseline: Dedicate one dimension per feature. Capacity: exactly $n$ features. If features are sparse, most dimensions are zero most of the time — wasteful.
Superposition alternative: Represent feature $i$ as unit vector $\mathbf{v}_i$ , tolerating small overlap $|\langle \mathbf{v}_i, \mathbf{v}_j \rangle| \le s$ for $i \neq j$ (an "almost-orthogonal" set).
How many almost-orthogonal vectors fit? This is the spherical-code / Johnson–Lindenstrauss question. The key, correct fact is: $m \;\lesssim\; \exp\!\big(c\, s^2\, n\big)$ for a fixed maximum overlap $s$ . That is, the number of vectors you can pack while keeping overlaps below $s$ grows exponentially in $n$ (for fixed $s$ ), not linearly. This is why a model with only $n$ dimensions can encode far more than $n$ features.

Why exponential, not $n/s^2$ ? Random unit vectors in $\mathbb{R}^n$ have pairwise overlap concentrated around $0$ with standard deviation $\approx 1/\sqrt{n}$ . The probability that a random pair exceeds overlap $s$ decays like $\exp(-c\,s^2 n)$ . A union bound over $\binom{m}{2}$ pairs stays small as long as $m \lesssim \exp(c\,s^2 n)$ — hence exponential capacity.
Sparsity is what makes it usable: Packing many vectors is geometrically possible, but decoding them under interference requires the active features to be few. If only $k \approx m p$ features are active at once, the total interference on any feature is $\sim s\sqrt{k}$ (random-sign accumulation). Keeping this below the signal ( $\approx 1$ ) requires $p$ small — so sparser features → more usable superposition.

Formula When Superposition Is Decodable (Toy Model)

Features stay separable as long as the accumulated interference is below the signal: $s\,\sqrt{m\,p}\;\lesssim\;\text{SNR}_{\min}$ Rearranging for the tolerable feature count: $m \;\lesssim\; \frac{\text{SNR}_{\min}^{\,2}}{p\,s^{2}}$

Read it carefully:

Interference scales as $s\sqrt{k}$ with $k=mp$ active features (random-walk accumulation), so the SNR appears squared and the overlap $s$ appears squared.
Dense features ( $p \to 1$ ): interference dominates, effective capacity collapses toward $n$ (no useful superposition).
Sparse features ( $p \to 0$ ): the denominator shrinks, so the decodable feature count grows large — heavy, usable superposition.
The geometric ceiling ( $m \lesssim e^{c s^2 n}$ ) is separate and much larger; sparsity, not geometry, is the binding constraint in practice.

Why Neurons Become Polysemantic

A polysemantic neuron activates for multiple unrelated features. This is a consequence of superposition.

Mechanism:

Model learns feature vectors $\{\mathbf{v}_1, \ldots, \mathbf{v}_m\}$ in $n$ -dimensional space
Neuron $j$ has activation $a_j = \sum_i f_i \cdot v_{ij}$ where $f_i$ is feature presence
If features 3, 17, and 42 all have a component along neuron $j$ 's axis, that neuron responds to all three

Example: An LM neuron might activate for:

The word "apple" (fruit feature)
Corporate tech mentions (Apple Inc. feature)
The color "red" (apple-color feature)

All three use overlapping subspaces → one neuron participates in encoding all three.

Worked example Worked Example: Toy 2D Superposition

Setup: 2 dimensions (2D space), want to store 4 binary features.

Feature vectors: Place at angles $0°, 60°, 120°, 180°$ : $\mathbf{v}_1 = \begin{bmatrix}1\\0\end{bmatrix}, \quad \mathbf{v}_2 = \begin{bmatrix}0.5\\0.866\end{bmatrix}, \quad \mathbf{v}_3 = \begin{bmatrix}-0.5\\0.866\end{bmatrix}, \quad \mathbf{v}_4 = \begin{bmatrix}-1\\0\end{bmatrix}$

Activation for feature 2 only: $\mathbf{a} = 1 \cdot \mathbf{v}_2 = \begin{bmatrix}0.5\\0.866\end{bmatrix}$ Neuron 1 activates at 0.5, neuron 2 at 0.866.

Interference test: If features 2 and 3 both active: $\mathbf{a} = \mathbf{v}_2 + \mathbf{v}_3 = \begin{bmatrix}0\\1.732\end{bmatrix}$

Can we decode? Project onto $\mathbf{v}_2$ : $\mathbf{a} \cdot \mathbf{v}_2 = 0 \cdot 0.5 + 1.732 \cdot 0.866 = 1.5$ (expect 1, got 1.5 → 50% interference).

Why step-by-step:

Used 2 dimensions to store 4 features (2× compression)
Overlap $s = \cos(60°) = 0.5$ causes errors
But if features are sparse (rarely co-occur), the scheme works!

How Circuits and Superposition Interact

Key relationship:

Circuits are the algorithm (the computational graph)
Superposition is the storage format (how features are packed)

A circuit might implement "detect edges then classify cat." The "edge" features might be superposed across 50 neurons, and the "cat" feature might be superposed with "dog" and "fox." The circuit uses specific linear combinations of those neurons to extract the right features.

Experimental Evidence

Induction heads (Anthropic, 2022): Identified repeating-token circuits formed by cross-layer composition of previous-token heads (earlier) and induction heads (deeper). Ablation strongly degrades in-context copying.
Superposition in toy models (Elhage et al., 2022): Trained ReLU networks on sparse features. Found networks reliably learn many more features than dimensions via superposition when features are sufficiently sparse.
Polysemanticity in vision models (Olah et al., 2020): Neurons in InceptionV1 respond to multiple unrelated concepts (e.g., "car" + "dog face"). Sparse autoencoders disentangle these into monosemantic features.

Recall Explain to a 12-Year-Old

Imagine your brain has to remember 1,000 different types of Pokemon, but you only have 100 storage boxes. You can't fit one Pokemon per box!

Solution: You notice that you rarely see all Pokemon at once—maybe only 10 appear in any battle. So you create a smart labeling system: "Box 47 stores a bit of Pikachu, a bit of Charizard, and a bit of Mewtwo." When Pikachu appears, you look in Box 47 (and Boxes 12, 89) and combine the clues to recognize Pikachu. This is superposition: storing more things than you have space by using overlap (because things don't all appear together).

Circuits are like your battle strategies: "If I see Pikachu, use Water-type." They're the rules you follow. The storage trick (superposition) just helps you remember more Pokemon so your strategies can be smarter!

Connections

6.3.01-Feature-visualization: Circuits process features; superposition encodes them
6.3.05-Attention-head-interpretability: Attention heads are circuit components in Transformers
6.3.08-Concept-activation-vectors: CAVs probe directions; superposition means multiple concepts per direction
4.2.03-Sparse-autoencoders: Used to disentangle superposed features
2.1.02-ReLU-and-activation-functions: Nonlinearity aids feature recovery under interference (but superposition itself is linear compression)

Flashcards

#flashcards/ai-ml

What is a neural network circuit?

A subgraph of the network (neurons + weighted connections) that implements a coherent, interpretable computational algorithm. Ablating it causally breaks the corresponding behavior.

What is superposition in neural networks?

When a model represents more features than dimensions by storing features as nearly-orthogonal (but not perfectly orthogonal) directions. It is fundamentally linear compression:

m > n

features in

n

dimensions.

Why does sparsity enable usable superposition?

Because only a few features are active at once, the accumulated interference

\sim s\sqrt{mp}

stays below the signal, so features remain decodable. Sparser features → more usable superposition.

Roughly how many almost-orthogonal vectors fit in

\mathbb{R}^n

for fixed overlap

s

Exponentially many:

m \lesssim \exp(c\,s^2 n)

. The geometric ceiling grows exponentially in

n

, so it is rarely the binding constraint — sparsity is.

What is a polysemantic neuron?

A neuron that activates for multiple unrelated features. It's a consequence of superposition: multiple feature vectors have components along that neuron's axis.

How do you identify a circuit?

(1) Observe behavior, (2) Hypothesize which neurons/heads are involved, (3) Ablate (zero out) those components, (4) Check if behavior breaks. Circuits are defined by causal necessity, not correlation.

Where do induction heads live in a Transformer, and why?

In middle/deep layers (e.g. layers 5, 7, 9 in GPT-2), deeper than the previous-token heads they read from — cross-layer composition requires the writer (earlier) to run before the reader (later).

Does superposition require a nonlinear activation?

No. Superposition is linear compression (

m>n

via a linear map, like random projections). Nonlinearity helps recover/denoise features, but is not needed to store them in superposition.

How are circuits and superposition related?

Circuits are the algorithm (computational graph); superposition is the storage format (how features are compressed). Circuits operate on linear combinations of superposed features.

Why can't you identify circuits by activation magnitude alone?

Because (1) circuits are about causal paths, not magnitude, and (2) superposition means the same neuron participates in multiple circuits. Must use ablation to test causal necessity.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho yaar, core idea yeh hai ki hum aksar sochte hain ki neural network ka har ek neuron ek specific cheez seekhta hai — jaise ek neuron "cat" detect karta hai, dusra "dog". Lekin reality mein aisa nahi hota. Network ek city ki tarah hai jahan multiple bus routes ek hi road share karte hain. Matlab, ek hi neurons ke group mein bahut saari features encode ho sakti hain — alag-alag activation patterns ke through. Isko superposition kehte hain: jab network ke paas features zyada hain aur dimensions kam, tab woh features ko "nearly orthogonal" directions mein pack kar deta hai taaki sab kuch thodi si jagah mein fit ho jaye. Aur circuits woh specific paths hain layers ke through jo koi ek meaningful algorithm implement karte hain, jaise "curved line detect karna" ya "ek token ko copy karna".

Ab yeh matter kyun karta hai? Socho, agar sirf 100 neurons 1,000 features encode kar sakte hain clever overlapping se, toh network basically compression kar raha hai — bahut efficient storage. Circuits samajhne se pata chalta hai ki network kya compute kar raha hai, aur superposition samajhne se pata chalta hai ki woh itni saari cheezein itni chhoti jagah mein kaise store karta hai. Example mein induction circuit dekho: ek "previous-token head" (early layer) tag karta hai ki har position se pehle kya aaya, phir ek "induction head" (deeper layer) us information ko padhkar pattern match karta hai — jaise "Mary ke baad John aaya tha" — aur next prediction copy kar deta hai. Isliye dono heads alag layers mein hote hain, kyunki jo head padhta hai woh baad mein chalna chahiye jo likhta hai.

Yeh sab interpretability ke liye super important hai. Agar hum verify karna chahte hain ki koi circuit sach mein kaam kar raha hai, toh hum usse ablate (zero out) kar dete hain — agar behavior tut jaata hai, matlab woh circuit causal hai, sirf coincidence nahi. Yeh technique AI ko "black box" se "understandable system" banane ki taraf le jaati hai, jo safety aur trust ke liye zaroori hai. Toh basically, circuits + superposition milkar humein andar jhaankne ka nazariya dete hain — ki yeh giant models actually andar se kaise sochte aur kaam karte hain.

What Are Circuits?

Why Circuits Form: From First Principles

What Is Superposition?

Deriving Superposition Capacity from Geometry

Why Neurons Become Polysemantic

How Circuits and Superposition Interact

Experimental Evidence

Connections

Flashcards

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Interpretability & Explainability

Connections