Interpretability & Explainability

Question 1 (12 marks)

(a) (3 marks) $\phi_i = \sum_{S \subseteq N\setminus\{i\}} \frac{|S|!\,(n-|S|-1)!}{n!}\,\big[v(S\cup\{i\}) - v(S)\big]$

• 1 mark: sum over subsets excluding $i$.
• 1 mark: correct weight $\frac{|S|!(n-|S|-1)!}{n!}$.
• 1 mark: marginal contribution term $v(S\cup\{i\})-v(S)$.

(b) (5 marks) The weight equals $1/n!$ times the number of orderings of the $n$ features in which the members of $S$ precede $i$ and the rest follow. Equivalently, $\phi_i = \frac{1}{n!}\sum_{\pi}[v(P_i^\pi\cup\{i\})-v(P_i^\pi)]$ where $P_i^\pi$ is the set of predecessors of $i$ in permutation $\pi$. (2 marks — permutation formulation.) Summing over $i$: for a fixed permutation $\pi$, the sum $\sum_i [v(\text{prefix up to and incl } i)-v(\text{prefix before } i)]$ telescopes to $v(N)-v(\varnothing)$. (2 marks — telescoping.) Averaging over all $n!$ permutations of a constant gives $\sum_i\phi_i = v(N)-v(\varnothing)$. (1 mark.)

(c) (4 marks) For $n=3$, weights: $|S|=0 \to 2!/3!=1/3$; $|S|=1 \to 1!\cdot1!/3!=1/6$; $|S|=2 \to 2!\cdot0!/3!=1/3$. Marginal contributions of feature 1:

• $S=\varnothing$: $v(\{1\})-v(\varnothing)=10$, weight $1/3$.
• $S=\{2\}$: $v(\{1,2\})-v(\{2\})=40-20=20$, weight $1/6$.
• $S=\{3\}$: $v(\{1,3\})-v(\{3\})=50-30=20$, weight $1/6$.
• $S=\{2,3\}$: $v(\{1,2,3\})-v(\{2,3\})=100-70=30$, weight $1/3$.

$\phi_1 = \tfrac13(10) + \tfrac16(20) + \tfrac16(20) + \tfrac13(30) = \tfrac{10}{3}+\tfrac{20}{6}+\tfrac{20}{6}+10 = \tfrac{10}{3}+\tfrac{10}{3}+\tfrac{10}{3}+10 = 10+10 = 20.$ $\boxed{\phi_1 = 20}$ (2 marks correct method, 2 marks correct value.)

Question 2 (10 marks)

(a) (6 marks)

def lime_explain(f, x, num_samples=5000):
    Z = []                      # perturbed samples (interpretable rep)
    for _ in range(num_samples):
        z = sample_perturbation(x)       # e.g. mask features/superpixels
        Z.append(z)
    Zp = [to_original_space(z) for z in Z]
    y  = [f(zp) for zp in Zp]            # black-box predictions
    # proximity weight: closeness of z to x
    w  = [exp(-D(x, zp)**2 / sigma**2) for zp in Zp]   # kernel
    # fit sparse linear surrogate g minimizing weighted loss
    g = weighted_lasso_fit(Z, y, weights=w)   # local, interpretable
    return g.coefficients               # feature attributions

Marks: perturbation sampling (2), predictions on perturbations (1), exponential kernel weighting by proximity (1), weighted (sparse) linear fit (2).

(b) (4 marks — 2 each)

• Instability / non-determinism: random sampling + local fit ⇒ different runs give different explanations; the fit depends heavily on kernel width $\sigma$ (an arbitrary hyperparameter that defines "locality").
• Local-linearity failure: LIME assumes the decision surface is locally linear; near sharp/curved boundaries the linear surrogate misrepresents $f$, producing misleading or even sign-wrong attributions. (Also acceptable: perturbations produce off-manifold/unrealistic inputs where $f$ behaves undefined.)

Question 3 (10 marks)

(a) (6 marks) Global-average-pool the gradients of the class score w.r.t. each feature map to get importance weights: $\alpha_k^c = \frac{1}{Z}\sum_i\sum_j \frac{\partial y^c}{\partial A^k_{ij}}$ (2 marks: gradient of class score; 1 mark: global average pool over spatial dims, $Z$ = #pixels.) Combine maps weighted by importance and rectify: $L^c_{\text{Grad-CAM}} = \mathrm{ReLU}\!\left(\sum_k \alpha_k^c A^k\right)$ (2 marks: weighted sum of feature maps; 1 mark: ReLU.)

(b) (2 marks) ReLU keeps only features with a positive influence on the target class; negative contributions belong to other classes. Without it the map would mix in evidence against the class and highlight regions unrelated to (or suppressing) class $c$, blurring localization.

(c) (2 marks) Input-gradients are computed w.r.t. individual pixels through many nonlinear layers, so they are high-frequency and sensitive to tiny input changes (gradient saturation/shattering); Grad-CAM operates on coarse, spatially-smooth high-level feature maps, averaging out that noise.

Question 4 (12 marks)

(a) (4 marks) Superposition: a network stores more distinct features than it has neurons/dimensions by representing them as near-orthogonal directions in activation space rather than one-neuron-per-feature. (2) It works because features are sparse — few are active at once — so the interference (dot-product overlap) from simultaneously active features is rare and tolerable; the model trades small interference noise for representing many more features. (2)

(b) (5 marks) Encoder/decoder over-complete dictionary: $f = \mathrm{ReLU}(W_{enc}(x - b_{dec}) + b_{enc}), \qquad \hat{x} = W_{dec} f + b_{dec}$ Loss: $\mathcal{L} = \underbrace{\|x - \hat{x}\|_2^2}_{\text{reconstruction}} + \lambda \underbrace{\|f\|_1}_{\text{sparsity}}$ Marks: encoder with ReLU (1), decoder reconstruction (1), reconstruction MSE term (1), L1 sparsity penalty on codes (1), $\lambda$ tuning tradeoff / definitions of $W,b$ (1).

(c) (3 marks) Dictionary size $= 512 \times 16 = 8192$ features. (1) It is over-complete because the number of dictionary atoms (8192) exceeds the activation dimension (512). (1) This gives room to disentangle superposed features into separate, mostly monosemantic directions — solving polysemanticity so each learned feature is human-interpretable. (1)

Question 5 (10 marks)

(a) (5 marks)

1. Clean run: run the model on a prompt that produces the correct/target behavior; cache activations. (1)
2. Corrupted run: run a minimally-different prompt (e.g. altered subject) that breaks the behavior; record the (lower) metric. (1)
3. Patch: re-run the corrupted prompt but overwrite a specific activation (a layer/position/head) with the cached clean value. (1)
4. Metric: measure recovery of the target output (e.g. logit-diff restored). (1) A large recovery effect supports a causal claim: that component is sufficient (in this context) to carry the information driving the behavior — not merely correlated with it. (1)

(b) (5 marks) High probe accuracy shows POS is linearly decodable from the activations, but decodability ≠ usage: the information could be an incidental byproduct never read by downstream computation. (2) Confound: a sufficiently expressive probe can memorize / fit structure present in rich activations regardless of task relevance (probe capacity confound). (1) Control: use a control task (random-label version) and report selectivity = task accuracy − control accuracy; a high selectivity indicates the representation genuinely encodes POS rather than the probe just being powerful. (2) (Activation patching / causal intervention also accepted as a stronger control.)

Question 6 (6 marks)

(a) (3 marks) $x_{cf} = \arg\min_{x'} \; \underbrace{\big(f(x') - y'\big)^2}_{\text{validity}} + \lambda \, \underbrace{d(x, x')}_{\text{proximity}}$ Marks: loss pushing $f(x')$ to target $y'$ (1), proximity/distance penalty (1), correct combination with weight $\lambda$ (1).

(b) (3 marks — 1.5 each; any two)

• Sparsity / minimality: change few features — a user can act on "raise income" not on 40 simultaneous changes.
• Actionability / plausibility (on-manifold): don't recommend changing immutable/unrealistic features (e.g. age down, or off-distribution values); the advice must be feasible.
• (Also: robustness, diversity of multiple counterfactuals.)

[
  {"claim":"Shapley phi_1 = 20 for given value function",
   "code":"from itertools import combinations\nfrom math import factorial\nv={frozenset():0,frozenset({1}):10,frozenset({2}):20,frozenset({3}):30,frozenset({1,2}):40,frozenset({1,3}):50,frozenset({2,3}):70,frozenset({1,2,3}):100}\nn=3\nother=[2,3]\nphi=0\nfor k in range(len(other)+1):\n  for S in combinations(other,k):\n    S=frozenset(S)\n    w=factorial(len(S))*factorial(n-len(S)-1)/factorial(n)\n    phi+=w*(v[S|{1}]-v[S])\nresult=abs(phi-20)<1e-9"},
  {"claim":"Efficiency: sum of Shapley values equals v(N)-v(empty)=100",
   "code":"from itertools import combinations\nfrom math import factorial\nv={frozenset():0,frozenset({1}):10,frozenset({2}):20,frozenset({3}):30,frozenset({1,2}):40,frozenset({1,3}):50,frozenset({2,3}):70,frozenset({1,2,3}):100}\nn=3\ndef phi(i):\n  others=[j for j in [1,2,3] if j!=i]\n  tot=0\n  for k in range(len(others)+1):\n    for S in combinations(others,k):\n      S=frozenset(S)\n      w=factorial(len(S))*factorial(n-len(S)-1)/factorial(n)\n      tot+=w*(v[S|{i}]-v[S])\n  return tot\nresult=abs(sum(phi(i) for i in [1,2,3])-100)<1e-9"},
  {"claim":"SAE dictionary size = 512*16 = 8192",
   "code":"result=(512*16)==8192"}
]