Exercises — Importance of interpretability

The parent note lists five reasons interpretability matters. These exercises invoke them by number, so here they are stated in one place:

1. Trust and Adoption — humans must verify the model's reasoning matches domain expertise before acting on it.
2. Debugging and Model Improvement — interpretability reveals what the model actually learned (e.g. a spurious shortcut), which high accuracy can hide.
3. Regulatory and Legal Compliance — laws require you to articulate, contest and audit automated decisions.
4. Bias Detection and Fairness — interpretability shows which features carry historical discrimination, not just that bias exists.
5. Safety in High-Stakes Domains — you must prove the reasoning is sound (autonomous vehicles, medical devices), not merely usually correct.

Recall the parent definitions: Global = understanding the entire model logic across all inputs; Local = understanding one single prediction.

• (a) Global — "across all patients" is the whole-model view.
• (b) Local — "for this patient" is a single prediction.
• (c) Global — "influences every prediction" is model-wide behaviour.

Interpretability (also called model transparency). It answers "Why did the model make this prediction?" and "How does it work?"

What we do: substitute each row into the formula. Why: the whole point of the parent's lesson is that accuracy alone is the wrong scoreboard in regulated, high-stakes settings.

Black box: $0.90-0.3(0.40)-0.4(0.30)-0.2(0.50)=0.90-0.12-0.12-0.10=\mathrm{0.56.}$ Interpretable: $0.85-0.3(0.10)-0.4(0.05)-0.2(0.10)=0.85-0.03-0.02-0.02=\mathrm{0.78.}$ Interpretable wins, $0.78>0.56$, even though it is 5 accuracy points worse. The penalties for risk, compliance and distrust erase the black box's raw accuracy edge.

$\Delta =|0.72-0.51|=\mathrm{0.21.}$ Demographic parity would require $\Delta =0$ (equal approval rate across groups). Since $\Delta =0.21\ne 0$ (a 21 percentage-point gap) on otherwise comparable applicants, parity is violated — a fairness red flag. The inequality holds, so equal treatment fails.

(a) What it learned: the rule if snow → wolf. Because the training photos had wolves in snow and huskies on grass, snow was a shortcut perfectly correlated with the label — a spurious correlation. The model latched onto the easy signal (bright white background) instead of the animal — exactly the amber region in the figure.

(b) Why accuracy hid it: the test set came from the same biased distribution as training — wolves in snow, huskies in grass — so the shortcut kept working. High test accuracy only proves the model matches that distribution, not that it learned the intended concept. This is exactly why the parent warns: high accuracy on test data doesn't guarantee the right features.

(c) Which pillar: Pillar 2, Debugging and Model Improvement. Interpretability revealed what was learned, turning an invisible bug into a visible one. (See Post-hoc Interpretability Methods for how saliency maps are computed.)

(d) Fix: augment the data with diverse backgrounds — huskies in snow, wolves on grass — so the snow shortcut no longer predicts the label, forcing the model to attend to the animal.

The mechanism is a proxy. "Prior arrests" is not race, but it is statistically entangled with race because policing patterns produced more arrests in minority communities for the same underlying offense rate. So "arrests" secretly carries the historical discrimination that "race" would have carried directly.

Removing the protected attribute is therefore not enough — the bias re-enters through a correlated feature. This is why the parent stresses that fairness metrics alone don't show how bias enters; interpretability reveals which feature is the carrier. (Deeper treatment: Fairness in Machine Learning.)

Trust is a function of three inputs, and P is strong in only one:

• Accuracy: P wins (99 vs 88).
• Interpretability: Q wins decisively — it names features; P is a bare number.
• Alignment with domain knowledge: Q wins — cholesterol, BP, smoking are exactly what cardiologists already trust as risk factors, so doctors can verify the reasoning. P offers nothing to check against, so doctors fear it may ride a spurious correlation (the parent's "zip code ↔ race" worry).

Two of three arguments favour Q, and the two it wins are the ones that let a human validate before acting on a high-stakes decision. A single dominant accuracy input can't compensate for zero interpretability and zero verifiable alignment — so $f$ ranks Q higher. This mirrors the parent's line: an 85%-accurate interpretable model often beats a 99%-accurate black box in adoption.

Check each against the three triggers high_stakes, regulated, needs_trust:

• (a) Meme captions → low stakes, unregulated, low trust demand → favor accuracy (black box + optional post-hoc is fine).
• (b) Parole → high stakes and affects liberty, subject to legal challenge → favor interpretability.
• (c) Loan denial → regulated (Fair Credit Reporting Act / ECOA require stating principal reasons and proving non-discrimination) → favor interpretability.
• (d) Song recommender → low stakes, no regulation, mistakes are cheap → favor accuracy.

The rule is an OR — a single trigger flips you to interpretability. (b) and (c) each fire at least one trigger; (a) and (d) fire none.

Set up the inequality. Penalties are fixed: $0.3(0.10)+0.4(0.05)+0.2(0.10)=0.03+0.02+0.02=\mathrm{0.07.}$ So $\text{Value}=A-0.07$. Require $A-0.07\ge 0.75$: $A\ge \mathrm{0.82.}$ Interpretation: because its penalties are tiny (0.07 total), this interpretable model needs only 82% accuracy to satisfy a regulator demanding Value $\ge 0.75$. A black box carrying the Ex-2.1 penalties (0.34 total) would need $A\ge 1.09$ — impossible, since accuracy caps at 1. That is the mathematical statement of the parent's whole thesis: interpretability lowers the accuracy bar you must clear by shrinking the penalty terms.

Strategy = keep the accurate model, bolt on explanation + documentation.

Part 1 — Post-hoc local explanations. Attach attention/saliency maps so each scan's prediction highlights the region driving it (e.g., the actual lesion, not an image artifact).

• Pillar: Pillar 5, Safety in high-stakes domains — clinicians must confirm the model looked at real pathology, not a scanner watermark, exactly like the autonomous-car attention-map example.
• Vault: Post-hoc Interpretability Methods, Explainable AI (XAI) Techniques.

Part 2 — Global documentation of scope and limits. Ship a model card — a short standardized document stating the model's training population, intended use, known failure modes, and per-subgroup test performance.

• Pillar: Pillar 3, Regulatory / Legal compliance — the model card provides the audit trail and "meaningful information about the logic" regulators want, and lets you contest / review decisions.
• Vault: Model Cards and Documentation.

Bonus robustness note: since deep nets are vulnerable to Adversarial Examples, the saliency maps also serve as a sanity check — if the highlighted region is nonsensical for a confident prediction, that's a signal of an adversarial or corrupted input. This ties Pillar 2, Debugging into a safety-critical deployment.

Why this design is defensible: you satisfied the "show reasoning" demand without sacrificing the accuracy that motivated the deep net — the parent's recommended path for pure_prediction ... add post-hoc interpretability, upgraded for a regulated domain by adding the global documentation layer.

Disparity (demographic parity) is measured between which two things? ::: The two conditional group distributions $P(\hat{y}=1\mid a)$ and $P(\hat{y}=1\mid b)$ — group vs group, never group vs overall average.