Level 1 — RecognitionInterpretability & Explainability

Interpretability & Explainability

⏱ 20 minutes30 marksprintable — key stays hidden on paper

Chapter: 6.3 Interpretability & Explainability Level: 1 — Recognition (MCQ, Matching, True/False with justification) Time limit: 20 minutes Total marks: 30

Section A — Multiple Choice (1 mark each) — 10 marks

Q1. SHAP values are grounded in which mathematical framework?

(a) Bayesian inference
(b) Shapley values from cooperative game theory
(c) Gradient descent
(d) Markov decision processes

Q2. LIME explains a prediction by:

(a) Retraining the full model on new data
(b) Fitting a simple interpretable model locally around the instance
(c) Computing exact global feature importance
(d) Visualizing attention heads

Q3. Grad-CAM produces explanations by using:

(a) Gradients flowing into the final convolutional layer
(b) Input pixel occlusion only
(c) A separate probing classifier
(d) Sparse autoencoder features

Q4. "Superposition" in mechanistic interpretability refers to:

(a) Stacking multiple models together
(b) A network representing more features than it has neurons by overlapping them
(c) Adding noise to activations
(d) Averaging attention weights

Q5. A probing classifier is used to:

(a) Modify model weights during training
(b) Test whether information is linearly decodable from internal representations
(c) Generate counterfactual inputs
(d) Prune redundant neurons

Q6. Activation patching is primarily used to:

(a) Repair corrupted training data
(b) Establish causal roles of activations by swapping them between runs
(c) Compress model checkpoints
(d) Increase attention sparsity

Q7. Sparse autoencoders are applied to model activations mainly to:

(a) Reduce inference latency
(b) Decompose activations into more monosemantic, interpretable features
(c) Replace the attention mechanism
(d) Perform data augmentation

Q8. A counterfactual explanation answers the question:

(a) Which neurons fired most?
(b) What minimal change to the input would flip the prediction?
(c) How was the model trained?
(d) What is the global feature ranking?

Q9. Concept-based explanations (e.g., TCAV) explain predictions in terms of:

(a) Individual raw pixels
(b) Human-understandable high-level concepts
(c) Loss surface curvature
(d) Optimizer hyperparameters

Q10. A commonly cited limitation of attention visualization is that:

(a) It requires labeled data
(b) Attention weights do not always faithfully explain model outputs
(c) It cannot be computed for transformers
(d) It only works for images

Section B — Matching (1 mark each) — 8 marks

Q11–Q18. Match each method/term (left) to its correct description (right). Write the letter.

#	Term		Description
Q11	LIME	A	Highlights input regions via gradient magnitudes
Q12	SHAP	B	Local surrogate interpretable model
Q13	Saliency map	C	Additive feature attribution with game-theoretic fairness axioms
Q14	Grad-CAM	D	Studies internal algorithms/circuits inside networks
Q15	Mechanistic interpretability	E	Class-discriminative heatmap from conv-layer gradients
Q16	Circuits	F	Swaps activations to test causal effect
Q17	Activation patching	G	Subgraphs of features/weights implementing a behavior
Q18	Probing classifier	H	Auxiliary model decoding info from representations

Section C — True/False WITH Justification (1 mark T/F + 1 mark reason) — 12 marks

For each statement, state True or False and give a one-line justification.

Q19. "High interpretability is only relevant for regulated domains like healthcare and finance." (2 marks)

Q20. "SHAP guarantees that feature attributions sum to the difference between the prediction and the expected (baseline) prediction." (2 marks)

Q21. "Grad-CAM and LIME are both inherently model-agnostic and require no access to internal gradients." (2 marks)

Q22. "Superposition makes individual neurons harder to interpret because a single neuron can respond to many unrelated features." (2 marks)

Q23. "A probing classifier achieving high accuracy proves the model uses that information for its predictions." (2 marks)

Q24. "Counterfactual explanations must always change as few features as possible while remaining plausible." (2 marks)

End of paper.

Answer keyMark scheme & solutions

Section A — MCQ (1 mark each)

Q1. (b) — SHAP derives from Shapley values in cooperative game theory, treating features as "players." Why: the fairness axioms (efficiency, symmetry, dummy, additivity) uniquely determine attributions.

Q2. (b) — LIME fits a simple, interpretable surrogate (e.g., sparse linear model) on perturbed samples weighted by locality. Why: it approximates the decision boundary locally, not globally.

Q3. (a) — Grad-CAM weights final conv feature maps by the gradient of the target class. Why: gradients indicate which spatial features are important for that class.

Q4. (b) — Superposition = more features than neurons, stored as overlapping directions. Why: enables high representational capacity but causes polysemanticity.

Q5. (b) — Probing tests linear decodability of information from hidden states. Why: a trained auxiliary classifier reveals what info is present.

Q6. (b) — Activation patching swaps activations between clean/corrupted runs to find causal components. Why: it isolates which internal states cause an output change.

Q7. (b) — SAEs decompose dense activations into sparse, more monosemantic features. Why: sparsity encourages disentangled, human-interpretable directions.

Q8. (b) — Counterfactuals identify the minimal input change flipping the outcome. Why: they answer "what would need to change" actionably.

Q9. (b) — Concept-based methods (TCAV) use human concepts as explanation units. Why: more meaningful than raw features to humans.

Q10. (b) — Attention weights are not guaranteed faithful explanations. Why: attention can be manipulated without changing outputs; correlation ≠ causation.

Section B — Matching (1 mark each)

Q	Answer
Q11 LIME	B — local surrogate model
Q12 SHAP	C — additive game-theoretic attribution
Q13 Saliency map	A — gradient-magnitude input highlighting
Q14 Grad-CAM	E — class-discriminative conv heatmap
Q15 Mechanistic interp.	D — studies internal algorithms
Q16 Circuits	G — subgraphs implementing behavior
Q17 Activation patching	F — swaps activations for causal test
Q18 Probing classifier	H — decodes info from representations

Section C — True/False with Justification (2 marks each)

Q19. False (1 mark T/F) — Justification (1 mark): Interpretability matters broadly for debugging, trust, safety, bias detection, and scientific insight, not only regulated fields.

Q20. True — Justification: This is SHAP's efficiency (local accuracy) property: $f(x)=\phi_0+\sum_{i=1}^{M}\phi_i$ , so attributions sum to the gap between prediction and baseline $\phi_0$ .

Q21. False — Justification: LIME is model-agnostic (uses only I/O), but Grad-CAM needs internal gradients/feature maps, so it is not gradient-free.

Q22. True — Justification: Under superposition a neuron is polysemantic, activating for multiple unrelated features, obscuring single-neuron interpretation.

Q23. False — Justification: High probe accuracy shows information is present/decodable, not that the model causally uses it (correlation, not causation).

Q24. False (partly) — Justification: Minimality and plausibility are desirable objectives, not strict requirements; valid counterfactuals need not be minimal, and other criteria (actionability, diversity) also matter.

Marking note: award 1 mark for correct T/F and 1 mark for a justification matching the key's reasoning.

[
  {"claim": "SHAP efficiency: prediction equals phi0 plus sum of feature attributions", "code": "phi0, p1, p2, p3 = symbols('phi0 p1 p2 p3'); fx = phi0 + p1 + p2 + p3; result = simplify(fx - (phi0 + (p1+p2+p3))) == 0"},
  {"claim": "Attribution sum equals prediction minus baseline", "code": "phi0, p1, p2, p3, fx = symbols('phi0 p1 p2 p3 fx'); eq = Eq(fx, phi0 + p1 + p2 + p3); attribsum = fx - phi0; result = simplify(attribsum - (p1+p2+p3).subs(fx, phi0+p1+p2+p3)) == 0"},
  {"claim": "Number of MCQ questions in Section A is 10", "code": "n = 10; result = (n == 10)"},
  {"claim": "Total marks: 10 MCQ + 8 matching + 12 TF = 30", "code": "result = (10 + 8 + 12) == 30"}
]