Interpretability & Explainability

Question 1

(a) Efficiency proof — 6 marks

Write the Shapley value in ordering form. Let $\Pi$ be the set of all $n!$ permutations of $N$. For $\pi\in\Pi$, let $\mathrm{Pre}_i(\pi)$ be the set of players preceding $i$. Then

(2 marks — equivalence of weight $\frac{|S|!(n-|S|-1)!}{n!}$ to counting orderings where the $|S|$ predecessors and $n-|S|-1$ successors are permuted freely.)

Summing over $i$ and swapping order of summation:

(1 mark.)

For a fixed permutation $\pi$, the inner sum telescopes: adding players one at a time from $\emptyset$ to $N$ gives

(2 marks — telescoping is the key step.)

Hence $\sum_i\phi_i = \frac{1}{n!}\cdot n!\,\big(v(N)-v(\emptyset)\big) = v(N)-v(\emptyset)$. $\blacksquare$ (1 mark.)

(b) Linear model closed form — 5 marks

Under independence, $\mathbb{E}[f(X)\mid X_S=x_S] = b + \sum_{j\in S} w_j x_j + \sum_{j\notin S} w_j\,\mathbb{E}[X_j]$. (2) So $v(S) = \sum_{j\in S} w_j\,(x_j - \mathbb{E}[X_j])$. (1) The marginal contribution of $i$ is $v(S\cup\{i\})-v(S) = w_i(x_i-\mathbb{E}[X_i])$ — independent of $S$. (1) Since Shapley weights sum to 1,

Interpretation: attribution = weight × deviation of feature from its baseline mean — "input×(weight,centered)". (1)

(c) Superposition & JL — 6 marks

JL lemma: for $m$ points, distortion $\varepsilon\in(0,1)$, a random projection into

dimensions preserves all pairwise distances up to factor $(1\pm\varepsilon)$. (2 marks — statement + bound.)

Consequence: $m \sim e^{\varepsilon^2 n}$ near-orthogonal directions fit in $n$ dims — exponentially more concepts than neurons. (1)

Misleading single-neuron Shapley: if concept $u_a$ is spread across many neurons (superposition), no single neuron $i$ "is" the concept; $\phi_i$ splits credit across a distributed code, and a neuron carrying parts of several concepts yields an attribution that cannot be read as "concept $a$'s importance." The attribution is well-defined for the neuron but not for the human concept, because the concept→neuron map is many-to-many. (3 marks: distributed code + polysemantic neuron + interpretation gap.)

(d) LIME vs SHAP — 5 marks

Failure mode (any valid): with correlated/interacting features, LIME's locally-fit linear surrogate weights depend on the sampling kernel $\pi_x$ and perturbation distribution; a strongly interacting feature can receive near-zero surrogate weight if perturbations rarely toggle it jointly, whereas SHAP's game-theoretic averaging assigns it non-zero credit. (3) LIME may violate local accuracy/efficiency (surrogate intercept + weights need not sum to $f(x)-\mathbb{E}[f]$) and/or consistency. (2)

Question 2

(a) Grad-CAM derivation — 8 marks

• Neuron importance weight: $\alpha_k^c = \frac{1}{Z}\sum_{i,j}\frac{\partial y^c}{\partial A^k_{ij}}$ where $Z=H\cdot W$. This is global average pooling of the gradients: it measures how much the score $y^c$ increases per unit increase averaged over the whole feature map, giving a single scalar "how important is channel $k$." (3)
• $Z$ normalises so weights are comparable across maps of different spatial size (average, not sum). (1)
• Linear combination $\sum_k \alpha_k^c A^k$ forms a weighted sum of activation maps — a coarse heatmap at conv resolution, upsampled to input size. (2)
• Final $\mathrm{ReLU}$ keeps only features with positive influence on the class of interest (negative values correspond to evidence for other classes and would clutter localisation). (2)

(b) Reduction to CAM — 6 marks

Head: $y^c = \sum_k w^c_k\,\mathrm{GAP}(A^k) = \sum_k w^c_k \cdot \frac{1}{Z}\sum_{i,j}A^k_{ij}$. (2) Then

(2) So

Up to the constant $1/Z$, the Grad-CAM weight equals the classifier weight $w^c_k$ — exactly the CAM formulation. (2)

(c) Model-randomization sanity check — 6 marks

def parameter_randomization_test(model, x, saliency_fn):
    S0 = saliency_fn(model, x)          # explanation for trained model
    sims = []
    for layer in top_to_bottom(model.layers):
        randomize_weights(layer)        # cascading randomization
        S = saliency_fn(model, x)
        sims.append(similarity(S0, S))   # SSIM / rank correlation
    return sims

Pass criterion: as parameters are randomized, saliency should change substantially (similarity → low). If the map is invariant to randomized weights, it is essentially an edge/input detector, not model-explaining → fail. (3)

Probabilistic reasoning: a trustworthy explanation should be a function of the learned conditional $p_\theta(y\mid x)$; if it is invariant to $\theta$ it carries no information about the model's decision function, i.e. mutual information $I(\text{saliency};\theta)\approx 0$. Passing (high sensitivity to $\theta$) is necessary for the map to reflect what the model actually computes. (3)

Question 3

(a) Activation patching definition — 6 marks

Run corrupted input but replace the activation $h_{L,t}$ with the clean value $h_{L,t}^{\text{clean}}$; let resulting logit be $\ell_{\text{patched}}$. Normalised patching effect:

(3) $\mathrm{PE}=1$ means that activation alone restores the clean answer; $0$ means it is irrelevant.

Causal, not correlational: we intervene ($do$-operation) on an internal variable and measure the change in output while holding everything else at the corrupted state. Unlike attention weights or probe accuracy (which only correlate with a concept), patching establishes that the activation causes the behaviour. (3)

(b) Probing pitfall proof — 6 marks

Counterexample: let $h=(h_1,h_2)^\top$, output $y = \mathrm{sign}(h_1)$. Suppose concept $c = \mathrm{sign}(h_2)$ and $h_2$ is present in the representation (perhaps written by an earlier layer) but the readout weight on $h_2$ is $0$. (2) A linear probe $p(h)=\mathrm{sign}(h_2)$ achieves accuracy $a=1$ for $c$. (2) Yet $\partial y/\partial h_2 = 0$: intervening on $h_2$ (activation patching along $c$) leaves $y$ unchanged — zero causal effect. Thus decodability $\neq$ usage. Patching is stronger because it tests the causal path, not mere presence of information. (2)

(c) Counterfactual gradient — 6 marks

$f(x')=\sigma(z)$, $z=w^\top x'+b$, $\sigma'(z)=\sigma(z)(1-\sigma(z))$. (1)

(2) For $\|x-x'\|_1$: $\nabla_{x'}\|x-x'\|_1 = -\,\mathrm{sign}(x-x') = \mathrm{sign}(x'-x)$ (subgradient, componentwise; $0$ at $x'_j=x_j$). (1)

(1) Sparsity: the $\ell_1$ subgradient has constant magnitude $1$ toward $x_j$ regardless of distance, so a feature $j$ is only moved if the data-term gradient's magnitude on it exceeds the $\ell_1$ pull; features with small $|w_j|$ stay at $x'_j=x_j$. This is the same LASSO thresholding mechanism → few features change. (1)

[
  {"claim":"Shapley efficiency: for a 2-player game sum of phi equals v(N)-v(empty)",
   "code":