Interpretability & Explainability

Q1. (2 marks) — Any two of, 1 mark each:

• Trust / accountability: stakeholders can verify the model reasons acceptably.
• Debugging: find spurious correlations or failure modes.
• Regulatory / legal compliance (e.g. right to explanation).
• Safety in high-stakes domains (medicine, finance).
• Fairness / bias detection.

Why: Interpretability supports trust, debugging, compliance and safety — the standard motivations.

Q2. (4 marks)

• LIME (1): fits a simple local surrogate model (e.g. sparse linear) to perturbed samples around one instance to approximate the black box locally.
• SHAP (1): attributes the prediction to features using Shapley values from cooperative game theory.
• Difference (1): LIME is heuristic/local with no guarantees; SHAP has a principled game-theoretic foundation.
• Unique SHAP property (1): satisfies axioms — local accuracy/efficiency (attributions sum to prediction), consistency, missingness (LIME does not guarantee these).

Q3. (5 marks)

For $N=\{1,2\}$, the coefficient weights:

• $S=\varnothing$: weight $=\frac{0!\,1!}{2!}=\tfrac12$.
• $S=\{2\}$: weight $=\frac{1!\,0!}{2!}=\tfrac12$.

$\phi_1 = \tfrac12[v(\{1\})-v(\varnothing)] + \tfrac12[v(\{1,2\})-v(\{2\})]$ $= \tfrac12(4-0) + \tfrac12(10-2) = 2 + 4 = 6.$ (3 marks)

Similarly $\phi_2 = \tfrac12(2-0)+\tfrac12(10-4)=1+3=4.$ (1 mark)

Check: $\phi_1+\phi_2 = 6+4 = 10 = v(N)$. ✔ (efficiency) (1 mark)

Q4. (4 marks)

• Saliency map (2): the gradient of the output class score w.r.t. input pixels, $\left|\frac{\partial y_c}{\partial x}\right|$ — highlights pixels whose change most affects the class score.
• Grad-CAM difference (2): uses gradients flowing into a convolutional feature map to produce class-weighted importance of channels, giving a coarse class-discriminative heatmap at the spatial resolution of the feature map (not per-pixel); it is localized to objects rather than noisy pixel-level gradients.

Q5. (4 marks) — Any two, 2 marks each:

• Attention is not explanation: different attention distributions can yield the same output (weights not identifiable).
• Information can flow through residual/value paths, so weights don't capture full contribution.
• Multiple layers/heads mix information, so a single-layer weight is not the causal driver.
• Attention weights weakly correlate with gradient-based importance.

Q6. (4 marks)

• Definition (2): a simple classifier (often linear) trained on a model's internal representations/activations to predict some property (e.g. part-of-speech), testing whether that property is encoded.
• High accuracy tells you (1): the information is present / linearly decodable in the representation.
• Does NOT let you conclude (1): that the model uses that information for its task (correlation, not causal use); a powerful probe may extract info the model ignores.

Q7. (4 marks)

• Superposition (2): a network represents more features than it has neurons/dimensions by encoding features as (nearly) linear directions that overlap, so individual neurons are polysemantic (respond to multiple unrelated features).
• Relation (2): when features are sparse, the model can pack $k > n$ features into $n$ dimensions with tolerable interference, trading a small reconstruction error for more capacity.

Q8. (5 marks)

• Objective (2): learn an overcomplete, sparse decomposition of activations $x$ into interpretable, monosemantic features $h$, to disentangle superposition.
• Loss (2): $\mathcal{L} = \underbrace{\|x - \hat{x}\|_2^2}_{\text{reconstruction}} + \lambda \underbrace{\|h\|_1}_{\text{sparsity}}, \quad h = \text{ReLU}(W_e x + b_e),\ \hat{x}=W_d h + b_d.$
• Role of terms (1): reconstruction term forces faithful encoding of activations; $L_1$ term encourages few active features (sparsity), yielding interpretable directions; $\lambda$ trades off the two.

Q9. (4 marks)

• Technique (2): run the model on a clean input, then replace (patch) a specific internal activation with the value it took on a corrupted/alternative input, and measure the change in output.
• Causal question (2): which components (layers/heads/positions) causally mediate a given behavior — i.e., locating where task-relevant information is processed.

Q10. (4 marks)

• Definition (2): a counterfactual explanation states the smallest change to the input features that would flip the model's decision to a desired outcome ("if feature $X$ were $x'$ instead of $x$, the prediction would change").
• Desirable property (2, any one): proximity/minimality (small change), actionability (feasible/realistic changes), sparsity (few features changed), plausibility/validity (stays on data manifold).

[
  {"claim":"phi_1 = 6 for the given cooperative game", "code":"v={frozenset():0,frozenset({1}):4,frozenset({2}):2,frozenset({1,2}):10}; from sympy import Rational, factorial; N={1,2}; import itertools; def phi(i):\n    tot=Rational(0)\n    others=N-{i}\n    for r in range(len(others)+1):\n        for S in itertools.combinations(others,r):\n            S=set(S); w=Rational(factorial(len(S))*factorial(len(N)-len(S)-1),factorial(len(N))); tot+=w*(v[frozenset(S|{i})]-v[frozenset(S)])\n    return tot\nresult=(phi(1)==6)"},
  {"claim":"phi_2 = 4", "code":"v={frozenset():0,frozenset({1}):4,frozenset({2}):2,frozenset({1,2}):10}; from sympy import Rational, factorial; N={1,2}; import itertools; def phi(i):\n    tot=Rational(0)\n    others=N-{i}\n    for r in range(len(others)+1):\n        for S in itertools.combinations(others,r):\n            S=set(S); w=Rational(factorial(len(S))*factorial(len(N)-len(S)-1),factorial(len(N))); tot+=w*(v[frozenset(S|{i})]-v[frozenset(S)])\n    return tot\nresult=(phi(2)==4)"},
  {"claim":"Efficiency: phi_1 + phi_2 = v(N) = 10", "code":"v={frozenset():0,frozenset({1}):4,frozenset({2}):2,frozenset({1,2}):10}; from sympy import Rational, factorial; N={1,2}; import itertools; def phi(i):\n    tot=Rational(0)\n    others=N-{i}\n    for r in range(len(others)+1):\n        for S in itertools.combinations(others,r):\n            S=set(S); w=Rational(factorial(len(S))*factorial(len(N)-len(S)-1),factorial(len(N))); tot+=w*(v[frozenset(S|{i})]-v[frozenset(S)])\n    return tot\nresult=(phi(1)+phi(2)==10)"}
]