Mechanistic interpretability: Question bank — AI-ML deep dive

Before we begin, three plain-word anchors so no term below appears unearned:

True or false — justify

TF — "A neuron that fires only for cat images proves the network has a dedicated cat circuit."

False — a single firing neuron is a correlation, not a mechanism; mechanistic interpretability demands you show the causal algorithm (edges → curves → ears) via patching, not just find a neuron that lights up.

TF — "Because each attention head has its own output block

W_{O}^{(i)}

, the heads are fully independent modules."

False — the

W_{O}^{(i)}

are column-blocks of one shared matrix

W_{O}

; the additive-over-heads form is a re-slicing of a single concat-then-project, not

H

separate networks.

TF — "Superposition means the network is confused and representing features badly."

False — superposition is a deliberate efficient trade: packing

m > n

sparse features into

n

dimensions, accepting small interference in exchange for far more representational capacity.

TF — "If a neuron is polysemantic, the network cannot be interpreted."

False — the neuron is a bad unit, but the underlying features are still clean directions; tools like Sparse Autoencoders recover those directions from the polysemantic mess.

TF — "The logit lens works because the residual stream accumulates information additively across layers."

True — since

h_{L} = h_{0} + \sum_{l} Δ h_{l}

, an early

h_{l}

already holds a partial linear answer that the unembedding

W_{U}

can read out.

TF — "Ablating a component and seeing performance drop proves that component computes the specific behavior we care about."

False — ablation only shows the component matters; it could be a general-purpose part. Only patching (transplanting its clean activation into a corrupted run) shows what information it carries.

TF — "Attention patterns alone tell you the full algorithm a head implements."

False — the pattern shows where information is routed (position

i

attends to

j

), but what content is moved and how it's used depends on the value/output projections, so you still need patching to confirm the mechanism.

TF — "Post-hoc interpretability and mechanistic interpretability are the same goal with different names."

False — post-hoc methods explain which inputs drove a decision; mechanistic interpretability reverse-engineers the internal computational steps that produced it.

TF — "The

1/ d_{k}

factor changes which token gets attended to."

Mostly false — it rescales logit variance to keep softmax out of saturation; it changes how sharp the attention is, not the ordering of raw dot products, though sharper softmax can effectively change routing at the extremes.

Spot the error

Error — "

h_{l + 1} = h_{l} + MLP (h_{l} + Attn (h_{l}))

is the transformer layer."

It drops LayerNorm; the real pre-norm form normalizes what each sublayer reads —

Attn (LN_{1} (h_{l}))

and

MLP (LN_{2} (\cdot))

— while the stream itself accumulates raw sums. Omitting LN misrepresents where nonlinearity acts.

Error — "Superposition lets you store

m

features orthogonally in

n < m

dimensions."

You cannot have more than

n

mutually orthogonal directions in

n

dimensions; superposition works precisely by using non-orthogonal directions and relying on sparsity so few collide at once.

Error — "For an induction head, position

i

attends to position

i - 1

That describes a previous-token head. An induction head attends to the token after the previous occurrence of the current token — completing the pattern "A B … A → B".

Error — "The logit lens uses

W_{U} \cdot h_{l}

directly on the raw residual stream."

It applies LayerNorm first:

logits_{l} = W_{U} \cdot LN (h_{l})

, matching how the final layer reads the stream; skipping LN gives mis-scaled, misleading logits.

Error — "To measure a component's causal effect, replace clean activations with corrupted ones and see if it still succeeds."

The standard patch runs the corrupted input and injects the clean activation (

a_{j} \leftarrow a_{j}^{clean}

); if performance recovers, that component mediates the correct computation.

Error — "A circuit is any set of components with high total activation."

A circuit must be causally important and implement a coherent algorithm and be minimally sufficient — high activation is neither necessary nor sufficient.

Why questions

Why — "Why patch activations instead of just ablating?"

Ablation answers whether a component matters; patching answers what information it carries and where that information flows, by testing if transplanting it restores the behavior.

Why — "Why does the circuits hypothesis phrase things as sparse subgraphs?"

Because a full network is too tangled to name; the empirical bet is that the useful computation concentrates in small, mostly-separable subgraphs that each do one interpretable job.

Why — "Why is the residual stream central to mechanistic interpretability?"

Every sublayer both reads from and writes to it additively, so it acts as a shared communication channel — features written early can be read many layers later, making linear readouts (logit lens, probes) meaningful.

Why — "Why does sparsity make superposition possible?"

If each feature is active with probability

s

, the expected number active at once is

\approx m s

; when

m s < n

, collisions are rare enough that interference stays small, so you can pack

m > n

features.

Why — "Why decompose attention additively over heads?"

Because empirically distinct heads perform distinct, nameable operations (previous-token, induction, duplicate-token), and the additive form lets you attribute output contributions head-by-head.

Why — "Why can't we just read individual neurons to understand the model?"

Superposition forces neurons to be polysemantic — one neuron encodes fragments of several unrelated features — so the neuron basis is not the feature basis; you must recover the true directions.

Edge cases

Edge — "What happens to the logit lens in very early layers (near

h_{0}

)?"

It typically produces near-uniform or embedding-driven predictions, because little computation has accumulated yet; the readout only becomes coherent once enough

Δ h_{l}

terms have been added.

Edge — "What if features are perfectly dense (every feature active every token,

s \to 1

)?"

Superposition collapses —

m s \neq < n

, interference is constant, and the network is forced back toward one-dimension-per-feature, so it can represent at most

n

features cleanly.

Edge — "What if patching a component doesn't recover performance even though ablating it hurt?"

The component matters but isn't sufficient alone; the needed computation is likely distributed across several components, so you must patch them jointly to see recovery.

Edge — "What if two supposedly separate circuits share components?"

That's expected under superposition and reuse; minimality is about sufficiency, not exclusivity, so a head can legitimately participate in more than one circuit's algorithm.

Edge — "What if a head's attention pattern is uniform (attends everywhere equally)?"

It's likely acting as an averaging/aggregation or a near-null head rather than routing specific information; the flat pattern signals no selective copy is happening, so look to MLPs for the computation.

Edge — "What happens under superposition when many features fire at once on one token?"

Interference spikes — the non-orthogonal directions overlap and corrupt each other's readout — which is exactly the cost the sparse-coding objective is minimizing.

Recall One-line self-check

Can you state, in a sentence each, ablation vs patching, superposition vs orthogonality, and previous-token vs induction head? ::: Ablation = does it matter; patching = what info flows. Orthogonality = one dim per feature (capped at $n$ ); superposition = non-orthogonal dirs exploiting sparsity ( $m > n$ ). Previous-token attends $i \to i - 1$ ; induction attends to the token after the prior copy of the current token.

Related deep material: Attention Mechanisms · Sparse Autoencoders · Transformer Architecture · Feature Visualization · Probing Classifiers · Model Editing