Circuits and superposition: Question bank — AI-ML deep dive

Before you start, the symbols we lean on constantly. Meet them once here so no line below surprises you.

Definition The symbols used on this page

$n$ = number of dimensions = how many coordinate slots (neurons) the activation vector has.
$m$ = number of features = how many distinct human-meaningful things the model wants to store. Superposition is the regime $m > n$ .
A feature vector $v_{i}$ = the fixed direction (a unit column vector in $n$ -dimensional space) along which feature $i$ is written. See the arrows in the figure below.
$s$ = overlap magnitude = the largest inner product $∣ ⟨ v_{i}, v_{j} ⟩ ∣$ we tolerate between two different feature directions. $s = 0$ means perfectly perpendicular; small $s > 0$ means "almost perpendicular".
$p$ = feature activation probability = the chance any single feature is switched on in a given input (its sparsity). Small $p$ = rarely active.
$k$ = number of simultaneously active features = how many of the $m$ features fire at once. Because each of the $m$ features fires independently with probability $p$ , the expected count is $E [k] = m p$ , and the true $k$ concentrates tightly around $m p$ when $m$ is large — that is the sense in which $k \approx m p$ (a mean, not an exact identity).
$SNR$ = signal-to-noise ratio = (the true value of the feature we want to read) ÷ (the accumulated interference from other active features). $SNR_{m i n}$ = the smallest $SNR$ at which decoding still succeeds — the threshold where signal just beats noise. Decoding works while $SNR ≳ SNR_{m i n} \approx 1$ .

Two more words: a feature is one thing the network represents; a dimension is one coordinate axis. Superposition is packing more features than dimensions by giving each feature its own direction rather than its own axis.

Picture the whole idea first. Figure 1 shows four features living as directions in a 2D plane — more arrows than the two axes could hold orthogonally; its caption explains the axes and the overshoot when two features are co-active. Figure 2 shows why interference stays small: random directions in high dimensions are nearly perpendicular (left panel), and the crosstalk piles up like a drunkard's random walk, not a straight sum (right panel).

Recall Why random directions are nearly perpendicular (the concentration step)

Where does $σ \approx 1/ n$ come from? Take one fixed unit vector and a second random unit vector $u$ with independent coordinates of equal variance. Their inner product $X = ⟨ v, u ⟩$ is a sum of $n$ roughly-independent zero-mean terms. Summing $n$ independent zero-mean pieces gives a total whose standard deviation grows like $n$ ; after normalising $u$ to unit length (dividing by $\approx n$ ) the spread of $X$ shrinks to $σ \approx 1/ n$ . So the typical overlap between two random directions collapses toward $0$ as $n$ grows — that is concentration of measure. Where does the Gaussian tail come from? By the central limit theorem that normalised sum is approximately Gaussian with standard deviation $1/ n$ . A Gaussian's tail past $s$ standard-deviation-units behaves like $e^{- (units)^{2} /2}$ ; here $s$ corresponds to $s / σ = s n$ units, so $Pr (∣ X ∣ > s) ≲ e^{- (s n)^{2} /2} = e^{- s^{2} n /2}$ . Writing this as $e^{- c s^{2} n}$ just packages the fixed factor $1/2$ (and CLT slack) into the constant $c$ — $c$ is that fixed positive number from the Gaussian tail; it does not depend on $n$ , $m$ , or $s$ .

Recall From the tail to exponential capacity (the algebra, step by step)

Assumption (state it or the claim is misleading): the $m$ feature vectors are drawn as random, isotropic unit vectors in $R^{n}$ .

Each single pair is "too close" with probability $Pr (∣ X ∣ > s) ≲ e^{- c s^{2} n}$ (previous callout).
There are $(2 m) = \frac{m ( m - 1 )}{2} \leq \frac{m ^{2}}{2}$ pairs. A union bound (the chance of any bad event is at most the sum of each event's chance) gives $Pr (some pair too close) ≲ \frac{m ^{2}}{2} e^{- c s^{2} n}$ .
Demand this stays below $1$ : $\frac{m ^{2}}{2} e^{- c s^{2} n} \leq 1$ .
Take logarithms of both sides (log turns products into sums and is monotone, so the inequality direction is preserved): $ln (\frac{m ^{2}}{2}) - c s^{2} n \leq 0$ , i.e. $2 ln m - ln 2 \leq c s^{2} n$ .
Solve for $ln m$ : $ln m \leq \frac{1}{2} (c s^{2} n + ln 2)$ . Exponentiating, $m \leq 2 e^{c s^{2} n /2}$ , i.e. $m ≲ e^{c s^{2} n /2}$ . The factor $\frac{1}{2}$ in the exponent is exactly the $2$ that appeared because we counted pairs ( $m^{2}$ ) and then took the square root when solving for $m$ . Capacity therefore grows exponentially in $n$ for fixed $s$ — under the random-isotropic assumption.

True or false — justify

A model with 512 neurons in a layer can represent at most 512 features.

False — with an orthogonal (one-axis-per-feature) scheme yes, but superposition stores features as nearly-orthogonal directions, so the geometric ceiling grows exponentially in the dimension

n

(as

m ≲ e^{c s^{2} n}

), not linearly.

Superposition requires a nonlinearity (like ReLU) to exist.

False — superposition is fundamentally linear compression; even a linear map can pack

m > n

features. Nonlinearity only helps you recover (denoise) features despite their interference.

If two features are stored along perfectly orthogonal directions, that is still superposition.

False — perfect orthogonality with

m \leq n

is just the ordinary "one direction per feature, no overlap" case. Superposition specifically means overlap

⟨ v_{i}, v_{j} ⟩ \neq = 0

so you can exceed

n

features.

A polysemantic neuron is evidence that the network failed to learn cleanly.

False — polysemanticity is the expected consequence of superposition: several feature directions each have a component on that neuron's axis, so it fires for all of them by design, not by mistake.

Ablating (zeroing) a circuit and seeing a behaviour break proves, on its own, that the circuit causes that behaviour.

Mostly false with a caveat — ablation is an intervention (stronger than correlation), but zeroing can cause off-target damage or push the network off its training distribution, so a broken behaviour might be collateral. You need controls: ablate random matched components, check the specific behaviour drops far more, and confirm the effect is not just "any perturbation hurts."

Denser features (each active more often) let you pack more usable features into the same dimensions.

False — it's the opposite: with

k \approx m p

active at once, interference scales as

s m p

, so as

p \to 1

interference dominates and usable capacity collapses back toward

n

. Sparsity is what makes superposition usable.

The number of almost-orthogonal directions you can fit grows like

n / s^{2}

False — under random isotropic vectors it grows exponentially, roughly

m ≲ exp (c s^{2} n)

; random pairs exceed overlap

s

with probability

≲ e^{- c s^{2} n}

, and a union bound over the

(2 m)

pairs stays safe up to exponentially many vectors.

An induction head can do its job even if it sits in an earlier layer than the previous-token head.

False — the induction head reads what the previous-token head wrote into the residual stream, so the writer must run first; the induction head must be deeper.

Spot the error

"We found neuron 3117 fires only for the word 'apple', so we've located the apple feature."

The claim of "only" is the error — under superposition a single feature spreads across many directions and a single neuron shares its axis with several features, so a lone neuron is rarely a clean feature; use tools like sparse autoencoders to disentangle.

"Since features are stored as directions, decoding feature

i

is impossible because other features leak onto its direction."

The leak is real but bounded: with sparsity only

k \approx m p

features are active, so total interference is

\sim s k

; if that stays below the signal (

SNR ≳ SNR_{m i n}

) the feature is still separable.

"The toy-model capacity

m ≲ SNR_{m i n}^{2} / (p s^{2})

and the geometric ceiling

m ≲ e^{c s^{2} n}

contradict each other."

They measure different things: the exponential is how many directions geometrically fit, while

SNR_{m i n}

(the minimum signal-to-noise ratio at which a feature is still readable, defined in the symbol box) sets how many stay decodable under interference. The decodability bound is far smaller, so sparsity — not geometry — is the binding constraint in practice.

"Superposition means each neuron encodes exactly one feature very efficiently."

Backwards — superposition means each feature is a direction spread over many neurons, and each neuron participates in many features (polysemantic). "One neuron = one feature" is the monosemantic ideal superposition breaks.

"An induction circuit just memorises that 'Mary' is followed by 'John'."

No — it implements an algorithm: find previous occurrences of the current token, look at what came after them, and copy that. It generalises to token pairs it never memorised, which is why it's a circuit, not a lookup.

"Circuits can't be studied on their own because everything in a network is entangled."

A circuit's defining properties are that it is localised, modular, and causal; you can isolate it, ablate it, and read its function — that modularity is precisely what makes circuit analysis possible.

Why questions

Why does the

SNR

(and the overlap

s

) appear squared in the decodable-capacity formula?

Because interference accumulates like a random walk over

k = m p

active features, giving magnitude

s k

; squaring both sides to solve

s m p ≲ SNR_{m i n}

for

m

produces the squares.

Why do two heads in an induction circuit have to live in different layers?

Cross-layer composition: the induction head consumes the "what preceded me" tag that the previous-token head wrote earlier, and information only flows forward through the residual stream, so the writer must precede the reader.

Why does gradient descent naturally produce circuits rather than a uniform tangle?

Only task-relevant paths get consistently reinforced gradients; irrelevant paths see near-zero gradient and stay small (pruning), so high-weight, interpretable subgraphs emerge along the useful paths.

Why is orthogonal one-feature-per-dimension coding "wasteful" when features are sparse?

If a feature is active only rarely (small

p

), its dedicated dimension sits at zero most of the time — you're paying a full coordinate for something rarely used, so sharing that coordinate via overlap is far more efficient.

Why does sparsity make interference tolerable even though the overlaps

s

haven't changed?

Interference depends on how many features are simultaneously active (

k \approx m p

, the expected number that fire), not on how many exist (

m

); low

p

keeps

k

small so the

s k

pile-up stays under the signal.

Why can a linear map already achieve superposition, yet we still care about activation functions?

Linearity suffices to pack features as overlapping directions, but recovering a clean feature under interference is a denoising step where a nonlinearity (e.g. ReLU thresholding away small crosstalk) earns its place.

Why do we insist on control ablations before claiming a circuit is causal?

Because zeroing components perturbs the network generally and can knock it off-distribution; only if the targeted ablation hurts the specific behaviour far more than random matched ablations do can we attribute the effect to that circuit rather than to generic damage.

Why does the capacity bound

m ≲ e^{c s^{2} n}

depend on the vectors being random, and what constant is

c

The bound uses a Gaussian tail for random directions;

c

is the fixed positive constant from that tail (it packages the factor

\frac{1}{2}

from the central-limit-theorem tail and does not depend on

n

m

, or

s

). A non-random, cleverly designed codebook can do somewhat differently (see the edge cases below).

Edge cases

What happens to superposition as

p \to 1

(every feature active every time)?

Interference saturates, the decodable count

m ≲ SNR_{m i n}^{2} / (p s^{2})

collapses toward

n

, and you effectively lose superposition — dense features force near-orthogonal, one-per-dimension coding.

What happens as

p \to 0

(features almost never co-occur)?

The denominator shrinks, so the tolerable feature count grows large — heavy, usable superposition, because active features rarely collide.

If you set the tolerated overlap

s = 0

, how many features fit?

Exactly

n

— zero overlap forces true orthogonality, and

R^{n}

holds only

n

mutually orthogonal directions, so there's no superposition at all.

Is the exponential capacity a property of any arrangement, or only random directions?

Only the random-isotropic case is what the Gaussian-tail argument covers. A worst-case (adversarial) arrangement — e.g. forcing many vectors to cluster near one direction — can violate the overlap bound with far fewer than

e^{c s^{2} n}

vectors, so the exponential figure is a typical-case (high-probability) statement, not a guarantee for every placement.

Can a deliberately designed (deterministic) codebook beat the random bound?

Sometimes — structured spherical codes (e.g. from error-correcting-code or lattice constructions) can hit the packing limits more tightly or evenly than random draws, but they still obey the same order-of-magnitude ceiling; the random argument just shows exponential capacity is achievable without cleverness, and trained networks land somewhere between random and hand-designed.

In the toy 2D example storing 4 features at

0°, 60°, 120°, 180°

, why does decoding one active feature give exactly the right value but two active features overshoot?

A single feature's projection onto its own unit direction returns its true coefficient, but when a second feature is co-active its non-zero overlap adds crosstalk onto the projection, inflating the estimate above the true 1 (see Figure 1's overlapping arrows).

What does a fully monosemantic layer look like, and is it common in trained nets?

Every neuron would fire for exactly one feature (no shared directions), which is the

m \leq n

, zero-superposition limit — rare in real over-parameterised-by-features models, which is why interpretability leans on tools that recover that clean picture.

Recall Quick self-check

Superposition possible without nonlinearity? ::: Yes — it is linear compression; nonlinearity only aids recovery. Binding constraint in practice — geometry or sparsity? ::: Sparsity; the geometric (exponential) ceiling is far larger. Is a polysemantic neuron a bug? ::: No, it's the expected consequence of packing many feature-directions. Does a single ablation prove causality? ::: No — you need control ablations to rule out off-target / off-distribution damage. Is the exponential bound guaranteed for every vector layout? ::: No — it is a typical-case (random) result; adversarial layouts can do worse.

Related deep dives: 6.3.01-Feature-visualization, 6.3.05-Attention-head-interpretability, 6.3.08-Concept-activation-vectors, 4.2.03-Sparse-autoencoders.