Interpretability & Explainability

---

Question 1 (12 marks)

Value function (with $v(\emptyset)=10$): $v(\{1\})=20,\ v(\{2\})=14,\ v(\{3\})=16,\ v(\{1,2\})=30,\ v(\{1,3\})=28,\ v(\{2,3\})=18,\ v(\{1,2,3\})=40.$

(a) Shapley value formula over marginal contributions across all $3!=6$ orderings.

Feature 1 marginal contributions:

• Order where 1 first: $v(1)-v(\emptyset)=20-10=10$ (2 orderings: 123,132)
• 1 after 2: $v(12)-v(2)=30-14=16$ (order 213)
• 1 after 3: $v(13)-v(3)=28-16=12$ (order 312)
• 1 after 2,3: $v(123)-v(23)=40-18=22$ (orders 231,321)

$\phi_1 = \frac{1}{6}[2(10)+16+12+2(22)] = \frac{1}{6}[20+16+12+44]=\frac{92}{6}=15.\overline{3}$ (3)

Feature 2 marginal contributions:

• 2 first: $v(2)-v(\emptyset)=14-10=4$ (2 orderings)
• 2 after 1: $v(12)-v(1)=30-20=10$
• 2 after 3: $v(23)-v(3)=18-16=2$
• 2 after 1,3: $v(123)-v(13)=40-28=12$ (2 orderings)

$\phi_2=\frac{1}{6}[2(4)+10+2+2(12)]=\frac{1}{6}[8+10+2+24]=\frac{44}{6}=7.\overline{3}$ (3)

Feature 3 marginal contributions:

• 3 first: $v(3)-v(\emptyset)=16-10=6$ (2 orderings)
• 3 after 1: $v(13)-v(1)=28-20=8$
• 3 after 2: $v(23)-v(2)=18-14=4$
• 3 after 1,2: $v(123)-v(12)=40-30=10$ (2 orderings)

$\phi_3=\frac{1}{6}[2(6)+8+4+2(10)]=\frac{1}{6}[12+8+4+20]=\frac{44}{6}=7.\overline{3}$ (3)

(b) $\phi_0+\phi_1+\phi_2+\phi_3 = 10 + \frac{92}{6}+\frac{44}{6}+\frac{44}{6} = 10 + \frac{180}{6} = 10+30 = 40 = f(1,1,1).$ ✓ Efficiency holds. (3)

---

Question 2 (12 marks)

(a) $\partial g/\partial x_1 = 6x_1 - x_2$; at $(2,2)$: $12-2=10$. $\partial g/\partial x_2 = -x_1$; at $(2,2)$: $-2$. $|10|>|-2|$ → feature $x_1$ ranked first. (4)

(b) Linear surrogate using local gradient: $w_1=10,\ w_2=-2$. Intercept so surrogate matches at $x_0$: $g(2,2)=3(4)-4=8$, so $w_0 = 8 - 10(2) - (-2)(2) = 8-20+4=-8$. $\hat g(x) = -8 + 10x_1 - 2x_2.$ (4)

(c) With large-neighbourhood sampling, LIME fits a least-squares line over a wide range where the $3x_1^2$ curvature dominates. Averaging the effective slope over a large $x_1$ range around 2 inflates the $x_1$ coefficient (secant slope $\ne$ tangent), while the $-x_1 x_2$ cross-term makes the fitted $x_2$ slope depend on the sampled $x_1$ values (could shrink toward $-\langle x_1\rangle$).

Numerical example: sampling $x_1\in\{1,3\}$ at $x_2=2$: values $g=1,\ 21$; secant slope $=(21-1)/2=10$ — but with $x_2$ varied, e.g. $x_2\in\{0,4\}$ at $x_1=2$ gives $g=12,4$, slope $=-4$ vs tangent $-2$. So large sampling changes magnitudes and can reorder features when curvature/interaction is strong. Both methods can therefore disagree because gradient is a point (tangent) property, LIME a regional (secant/averaged) property. (4)

---

Question 3 (12 marks)

(a) Procedure (6): Run the model on the corrupted input ("Colosseum...") to get corrupted activations, and separately run the clean input ("Tower...") caching all activations. Then re-run the corrupted forward pass but replace (patch) the residual-stream activation at layer $\ell$, at the position of the subject token, with the corresponding clean-run activation at that layer/position. Keep everything else corrupted. Read out the logit difference between the two candidate answers (e.g. logit("Paris") − logit("Rome")) at the final token. If patching this one activation restores the clean answer, that activation causally carries the subject information. (6)

• (2) correct direction of copy (clean→corrupted)
• (2) correct site (residual stream, layer $\ell$, subject token position)
• (2) correct readout metric (logit difference)

(b) Normalized effect: $M=\frac{L_{\text{patch}}-L_{\text{corr}}}{L_{\text{clean}}-L_{\text{corr}}}.$ $M=1$ → full causal recovery (patch restores clean behaviour); $M=0$ → no effect (patch changed nothing). (3)

(c) Interpretation: layer 4 activation at that token carries little causal info ($M=0.05$); by layer 8 the information needed to produce the answer has been assembled/moved there ($M=0.9$). (2) Confound: activation patching measures sufficiency of a single component in isolation; distributed/redundant representations or backup circuits mean a low score elsewhere may under-credit a component, and correlated (not causal) info can inflate a score — also, patching residual stream mixes contributions of all upstream heads, so layer 8's high score doesn't localize which head wrote it. (1)

---

Question 4 (12 marks)

(a) Cosines (all unit vectors, so $\cos=$ dot product):

• $d_1\cdot d_2 = (1)(-0.5)+(0)(\tfrac{\sqrt3}{2}) = -0.5$
• $d_1\cdot d_3 = -0.5$
• $d_2\cdot d_3 = 0.25 - 0.75 = -0.5$

All pairwise $\cos=-0.5$ (120° apart). (3) Because sparse features rarely co-activate, the network tolerates non-orthogonality: at any moment few features are on, so the interference from the $-0.5$ overlap rarely corrupts decoding; spreading 3 directions symmetrically at 120° minimizes worst-case interference in 2D. (2)

(b) Score each direction by dot product with $a=(0.5,0.87)$:

• $a\cdot d_1 = 0.5$
• $a\cdot d_2 = -0.25 + 0.87(0.866)=-0.25+0.753=0.503$
• $a\cdot d_3 = -0.25 - 0.753 = -1.003$

$d_2$ scores highest ($\approx0.503$), so concept $c_2$ is most likely active (note $a\approx d_2$ since $(-0.5,0.866)$ ≈ up to sign — actually $a=(0.5,0.87)$ closely matches $d_2$'s $y$; $d_2$ wins). (3)

(c) (i) The $L_1$ penalty enforces sparsity in the hidden code, so each activation is explained by few active latent units — this is what disentangles superposed features into separate, mostly monosemantic dictionary atoms (without it, an overcomplete AE just relearns dense/polysemantic directions). (2) (ii) Report e.g. feature interpretability / monosemanticity: for each learned latent, inspect its max-activating inputs and check they share one coherent concept — quantitatively, low activation entropy across concepts, high automated-interpretability score, or high specificity/selectivity (fires on one concept, silent otherwise). Also acceptable: reconstruction loss vs. L0 sparsity tradeoff. (2)

---

Question 5 (12 marks)

(a) $s = 0.1(40) - 8(0.5) - 1 + 0.5(2) = 4 - 4 - 1 + 1 = 0$?

Recompute: $4 - 4 = 0$; $0 - 1 = -1$; $-1 + 1 = 0$. So $s=0$. Since deny is $s<0$ and here $s=0\ge0$... Let's re-read: with credit-length 2, $s=0$ → approve. To ensure denial as stated, the intended value: $s=0$ is the boundary. Answer: $s=0$, exactly on boundary (approve by $\ge0$ rule). Award full marks for correct arithmetic $s=0$ and noting boundary. (2)

(Marker note: model output $s=0$; treat as boundary case.)

(b) Need $s\ge0$. Baseline contributions: $0.1\cdot\text{inc} - 8\cdot\text{dr} - 1 + 0.5\cdot\text{cl}$.

Solve each holding others at applicant values (inc=40, dr=0.5, cl=2), and using the fixed sum of the other terms.

• Income: $0.1\,I - 4 - 1 + 1 \ge 0 \Rightarrow 0.1I \ge 4 \Rightarrow I \ge 40$. New income $\ge 40$ (already at boundary; any increase, e.g. $I=40$ gives $s=0$). Minimal: $I\ge40$. (2)
• Debt-ratio: $4 - 8\,d - 1 + 1 \ge 0 \Rightarrow 4 - 8d \ge 0 \Rightarrow d \le 0.5$. New debt-ratio $\le 0.5$. Minimal decrease from 0.5: $d\le0.5$. (2)
• Credit-length: $4 - 4 - 1 + 0.5c \ge 0 \Rightarrow 0.5c \ge 1 \Rightarrow c \ge 2$. New credit-length $\ge 2$ years. (2)

(Since applicant sits exactly at $s=0$, all three are at their thresholds; a strict flip to $s>0$ needs any improvement: e.g. $I>40$, $d<0.5$, or $c>2$.) Full marks for the three inequalities/thresholds.

(c) With credit-length non-actionable, present the