Circuits and superposition: Worked examples — AI-ML deep dive

This is the hands-on companion to the parent topic. There we built the ideas: a circuit is a task-relevant path through the network, and superposition is packing more features than dimensions by using nearly-orthogonal directions. Here we grind through every kind of numeric situation the topic can throw at you.

Before we start, three plain-words reminders (nothing new is assumed):

The single decoding move we repeat everywhere is: to read off "how much of feature $i$ is present" in a mixed activation $a$ , compute $a \cdot v_{i}$ . Why this and not something fancier? Because the dot product is the projection — it asks exactly the question "how far does $a$ reach along the $v_{i}$ direction?", and that is the definition of "how present is feature $i$ ".

The scenario matrix

Every numeric question in this topic is one of these cells. The worked examples below are each tagged with the cell(s) they cover.

Cell	What makes it distinct	Covered by
A. Orthogonal store	Overlap exactly $0$ , decode is perfect	Ex 1
B. Single non-orthogonal feature	One feature active, small self-clean read	Ex 2
C. Two features interfere (same sign)	Overlaps add → over-read	Ex 3
D. Two features interfere (opposite sign)	Overlaps cancel → under-read / miss	Ex 4
E. Zero / degenerate input	No features active, or two identical directions	Ex 5
F. Sparsity capacity (formula)	Plug into $m ≲ SNR^{2} / (p s^{2})$	Ex 6
G. Geometric ceiling (formula)	Plug into $m ≲ e^{c s^{2} n}$	Ex 7
H. Polysemantic neuron read	One neuron, many features summed on its axis	Ex 8
I. Real-world word problem	Circuit / feature reasoning in a sentence	Ex 9
J. Exam twist	Sign trap — "the projection lies to you"	Ex 10

Ex 1 — Cell A: the orthogonal baseline

Forecast: guess before reading — will the decode be exactly right, or contaminated? (Answer: exactly right.)

Write the activation. $a = 3 v_{1} + (- 2) v_{2} = [3 - 2]$ . Why this step? The activation is always the weighted sum of feature arrows; that is the model's actual internal state.
Decode feature 1: $a \cdot v_{1} = 3 \cdot 1 + (- 2) \cdot 0 = 3$ . Why this step? Projection onto $v_{1}$ answers "how much feature 1?".
Decode feature 2: $a \cdot v_{2} = 3 \cdot 0 + (- 2) \cdot 1 = - 2$ .

Verify: overlap $v_{1} \cdot v_{2} = 0$ (perpendicular), so no cross-talk term exists. Read $3$ and $- 2$ match the true strengths exactly. This is the wasteful case: $n = 2$ dimensions hold only $m = 2$ features. See 2.1.02-ReLU-and-activation-functions for why nonlinearity later lets us clean up the non-orthogonal cases.

Ex 2 — Cell B: one non-orthogonal feature, self-read

Forecast: guess — is $v_{2} \cdot v_{2}$ equal to $1$ , or something less?

Activation: $a = 1 \cdot v_{2} = [0.5 0.866]$ . Neuron 1 reads $0.5$ , neuron 2 reads $0.866$ (red arrow in figure). Why this step? Even one feature spreads its strength across both neurons — this is already superposition-flavoured storage.
Self-decode: $a \cdot v_{2} = 0. 5^{2} + 0.86 6^{2} = 0.25 + 0.75 = 1.0$ . Why this step? A unit vector dotted with itself is its squared length $= 1$ ; that guarantees a clean self-read when nothing else is active.

Verify: $0.25 + 0.75 = 1.0$ . A lone feature on a unit vector always decodes to its true strength — interference only appears when a second feature joins.

Ex 3 — Cell C: two features, same-sign interference (over-read)

Forecast: true strength of feature 2 is $1$ . Will the projection read exactly $1$ , more, or less?

Sum the activation: $a = v_{2} + v_{3} = [0 1.732]$ . Why this step? When two features fire together the model only sees their sum — a single arrow pointing straight up (plum arrow in figure).
Compute the overlap $v_{2} \cdot v_{3} = (0.5) (- 0.5) + (0.866) (0.866) = - 0.25 + 0.75 = 0.5$ . Why this step? The overlap is the leak coefficient — it tells us how much feature 3 contaminates feature 2's read.
Decode feature 2: $a \cdot v_{2} = 0 \cdot 0.5 + 1.732 \cdot 0.866 = 1.5$ . Why this step? Projection onto $v_{2}$ ; but now $a$ contains feature 3 too.

Verify (split the read): $a \cdot v_{2} = 1 v_{2} \cdot v_{2} + 0.5 v_{3} \cdot v_{2} = 1 + 0.5 = 1.5$ . We expected $1$ but got $1.5$ : the positive overlap $0.5$ added noise. This is the parent note's cliff-hanger resolved — the interference is exactly the overlap $s = 0.5$ .

Ex 4 — Cell D: opposite-sign interference (under-read / miss)

Forecast: overlap of these two is negative. Will feature 2 read above or below its true $1$ ?

Overlap: $v_{2} \cdot v_{5} = (0.5) (0.5) + (0.866) (- 0.866) = 0.25 - 0.75 = - 0.5$ . Why this step? Negative overlap means the two arrows point somewhat opposite — feature 5 will subtract from feature 2's read.
Activation: $a = v_{2} + v_{5} = [10]$ .
Decode: $a \cdot v_{2} = 1 \cdot 0.5 + 0 \cdot 0.866 = 0.5$ . Why this step? Split it: $v_{2} \cdot v_{2} + v_{5} \cdot v_{2} = 1 + (- 0.5) = 0.5$ .

Verify: true strength $1$ , read $0.5$ — an under-read. Same magnitude of interference as Ex 3, opposite direction, because the sign of the overlap flipped. Together Ex 3 and Ex 4 show why interference is modelled as a random-sign walk in the parent's $s k$ formula: sometimes it adds, sometimes it cancels.

Ex 5 — Cell E: zero and degenerate inputs

Forecast: in (b), can the network ever tell feature $a$ apart from feature $b$ ?

(a) Zero input decode: $a \cdot v_{2} = 0$ . Why this step? If nothing fires there is nothing to read — the decode must return $0$ , our sanity floor.
(b) Sum degenerate features: $a = v_{a} + v_{b} = [20]$ .
(b) Decode feature $a$ : $a \cdot v_{a} = 2 \cdot 1 + 0 \cdot 0 = 2$ . Why this step? Overlap $v_{a} \cdot v_{b} = 1$ (identical arrows) — the maximum possible interference.

Verify: In (b), decoding $a$ and decoding $b$ both give $2$ ; there is no way to separate them because their overlap is $1$ , not $\approx 0$ . This is why superposition needs near-orthogonal ( $s$ small), not just "many" directions. Overlap $= 1$ is the failure limit; overlap $= 0$ (Ex 1) is the perfect limit; everything real lives between. Sparse-autoencoder methods (4.2.03-Sparse-autoencoders) exist precisely to unfold the in-between cases back toward Ex 1.

Ex 6 — Cell F: sparsity capacity formula

Forecast: guess an order of magnitude — hundreds? thousands?

Square the SNR: $SNR_{m i n}^{2} = 2^{2} = 4$ . Why this step? Interference accumulates as $s m p$ , so keeping it below the signal squares the SNR — the parent derived this.
Square the overlap: $s^{2} = 0. 1^{2} = 0.01$ . Why this step? Same random-walk reason: overlap enters squared.
Divide: $m ≲ \frac{4}{0.01 \times 0.01} = \frac{4}{0.0001} = 40000$ .

Verify: $40000 ≫ n = 100$ — heavy usable superposition, entirely because $p$ is tiny. Set $p = 1$ (dense) and $m ≲ 4/0.01 = 400$ ; the capacity collapses toward $n$ , exactly as the parent warned. Sparsity, not dimension count, is the lever.

Ex 7 — Cell G: the geometric ceiling formula

Forecast: bigger or smaller than the $40000$ from Ex 6?

Exponent: $c s^{2} n = 1 \times 0.01 \times 100 = 1$ . Why this step? The geometric bound counts how many near-orthogonal arrows fit, ignoring whether we can decode them.
Exponentiate: $m ≲ e^{1} \approx 2.718$ .

Verify: Here the numbers are chosen so the geometric ceiling ( $\approx 2.7$ , tiny because $s^{2} n$ is small) is below the decodability count — a reminder these are two separate bounds and the smaller one binds. Increase $n$ to $1000$ : exponent $= 10$ , $e^{10} \approx 22026$ , and geometry stops being the bottleneck. The lesson: geometry grows exponentially in $n$ , sparsity-decodability grows linearly in $SNR^{2} / (p s^{2})$ — for realistic large $n$ , decodability (sparsity) binds first.

Ex 8 — Cell H: reading a polysemantic neuron

Forecast: the neuron is a mush of features — will it be positive?

Sum active contributions on the $j$ axis: $a_{j} = 1 \cdot 0.4 + 0 \cdot (- 0.3) + 1 \cdot 0.6 = 1.0$ . Why this step? By the parent's formula $a_{j} = \sum_{i} f_{i} v_{ij}$ , a neuron's value is the sum of every active feature's component along its axis.
Interpret: a reading of $1.0$ came from two unrelated features (3 and 42), not one. Why this step? This is the mechanism of polysemanticity — see it firing tells you something is active, not what.

Verify: turn feature 17 on too: $a_{j} = 0.4 - 0.3 + 0.6 = 0.7$ — a different value from a different mix. The single scalar $a_{j}$ can never be inverted to a unique feature set. This is why 6.3.01-Feature-visualization on a raw polysemantic neuron shows a jumble of unrelated images.

Ex 9 — Cell I: real-world word problem (induction circuit)

Forecast: does ablation prove the head causes the behaviour, or merely correlates?

Trace the circuit. Previous-token head tags the second "Mary" with "what came before me". Induction head at layer 7 reads that tag, searches history for the earlier "Mary", and copies the token that followed it ("John"). Why this step? Ordering matters — the reader (layer 7) must run after the writer (layer 2), which is why they sit in different layers.
Compute the drop: $92% - 18% = 74$ percentage points. Why this step? The size of the drop measures how much of the behaviour flowed through this head.
Interpret causality. A near-total collapse under ablation means the head is causal, not incidental.

Verify: $92 - 18 = 74$ . A $74$ -point drop from removing one head is the causal fingerprint the parent demanded. Contrast: ablating an unrelated head would leave accuracy near $92%$ . This mirrors 6.3.05-Attention-head-interpretability, where the same ablate-and-measure test isolates head function.

Ex 10 — Cell J: the exam twist (the projection lies)

Forecast: feature 1 is genuinely absent. Will its decoded value be near $0$ , or dangerously large?

Confirm $v_{2}$ is unit length: $0. 9^{2} + 0.43 6^{2} = 0.81 + 0.190 = 1.000$ . Good. Why this step? A fair overlap comparison needs unit vectors.
Activation (only feature 2 on): $a = v_{2} = [0.9 0.436]$ .
Decode feature 1: $a \cdot v_{1} = 0.9 \cdot 1 + 0.436 \cdot 0 = 0.9$ . Why this step? Projection onto $v_{1}$ — the standard decode.

Verify: feature 1 is OFF, yet its decoded read is $0.9$ — almost a full "present" signal! The culprit is the huge overlap $s = 0.9$ between the two directions. This is the exam trap: a big projection does not prove a feature is present when directions are not near-orthogonal. It is the numeric reason superposition requires $s$ small, and the motivation for 6.3.08-Concept-activation-vectors and sparse dictionaries to define cleaner directions.

Recall Self-test

When two active features have overlap $s > 0$ , the decoded value of one is its true strength plus what? ::: Plus $s$ times the other feature's strength (same-sign adds, opposite-sign subtracts). Which quantity — geometry ( $e^{c s^{2} n}$ ) or sparsity ( $SNR^{2} / p s^{2}$ ) — usually binds capacity in practice? ::: Sparsity (the decodability bound); the geometric ceiling is far larger for realistic $n$ . A neuron reads $a_{j} = 0$ . Does that mean no feature is active? ::: No — active features with opposite-sign components on that axis can cancel to $0$ (see the degenerate/cancellation cases).