Worked examples — Importance of interpretability

---

The scenario matrix

Interpretability problems come in a small number of shapes. Think of the table below as a checklist: by the end of this page we will have worked one example per cell, so you never meet a case you have not seen.

Cell	Case class	What makes it tricky	Example that covers it
A	Trust decision (accuracy vs interpretability trade-off)	Two numbers pull opposite ways	Example 1
B	Debugging a spurious correlation	Model is "right" for the wrong reason	Example 2
C	Fairness / group disparity, both signs of the gap	Gap can favour group $a$ or group $b$	Example 3
D	Zero / degenerate input: no disparity, or perfect parity	The formula must not "invent" a problem	Example 4
E	Limiting value: accuracy $\to 100\%$ but stakes very high	High accuracy does not rescue a black box	Example 5
F	Local vs global interpretability mismatch	One prediction explained ≠ whole model understood	Example 6
G	Real-world word problem (credit / adverse action)	Translate a legal rule into a number	Example 7
H	Exam-style twist: proxy feature hidden inside "fair" features	Bias sneaks in through a correlated column	Example 8

Two tools recur, so we define them now, before use:

---

Example 1 — Cell A: the trust trade-off (a real number, not a vibe)

Forecast: Guess before reading — does the 5-point accuracy lead save Model X?

1. Compute Value(X). ${\text{Value}}_X=0.90-0.3(0.4)-0.4(0.5)-0.2(0.3)$. Why this step? We are not comparing accuracy alone; the whole point of the parent note is that accuracy is only the first term. ${\text{Value}}_X=0.90-0.12-0.20-0.06=\mathrm{0.52.}$
2. Compute Value(Y). ${\text{Value}}_Y=0.85-0.3(0.1)-0.4(0.05)-0.2(0.05)$. Why this step? Same yardstick applied to both — a fair comparison requires the same $\lambda$ weights. ${\text{Value}}_Y=0.85-0.03-0.02-0.01=\mathrm{0.79.}$
3. Compare. $0.79>0.52$, so the interpretable Model Y wins by a wide margin. Why this step? This answers the parent's claim numerically: in a regulated, high-stakes setting the penalties on a black box can dwarf a small accuracy edge.

Verify: Sanity check the signs — every penalty term is subtracted, and X has larger penalties in all three, so it must lose ground relative to its accuracy lead of only $0.05$. The lead ($0.05$) is smaller than the extra penalty X carries, so the flip is expected. ✓

---

Example 2 — Cell B: debugging a spurious correlation (Husky–Wolf, quantified)

Forecast: Will a dumb "snow → wolf" rule get close to the fancy 95%?

1. List the four outcomes. Wolf-on-snow: correctly "wolf". Wolf-on-grass: wrongly "husky". Husky-on-grass: correctly "husky". Husky-on-snow: wrongly "wolf". The figure below plots each case as a bar; its height is that case's share of the whole test set. The two magenta bars are the cases the rule gets right, the two orange bars are the cases it gets wrong — each bar is annotated with its exact height. Why this step? Accuracy is a count over cases; we must enumerate every case (this is the "cover all cases" rule).

2. Weight by frequency. Half the animals are wolves, half huskies. Read the two magenta bars off the figure ($0.40$ and $0.40$) and add them: ${\text{Acc}}_{\text{snow-rule}}=0.5(0.80)+0.5(0.80)=\mathrm{0.80.}$ Why this step? $80\%$ of each species sits on its "expected" background (the tall magenta bars at height $0.40$ each), so the rule is right $80\%$ of the time per species; average over the 50/50 mix.
3. Interpret. The snow rule alone reaches $0.80$. The extra $0.95-0.80=0.15$ comes from some real features — but $80\%$ of the score is fake. Why this step? This is exactly why interpretability catches the bug: high test accuracy hid a mostly-spurious model. Notice from the figure how small the orange "WRONG" bars ($0.10$ each) are — that is precisely why the bug stays hidden until a saliency tool exposes it.

Verify: Deploy on wolves in grass (the left orange bar, now the only kind of wolf). The snow-rule now says "husky" for every such wolf → accuracy on that slice collapses toward $0$, confirming the model did not learn the animal. ✓

---

Example 3 — Cell C: fairness gap, both signs (a gap between two FPRs)

Forecast: Guess the sign of $\Delta$ before computing.

1. Original. $\Delta =0.45-0.25=+0.20$. Why this step? We compare the two groups' FPRs (both conditioned on $y=0$), not group-vs-marginal — this is the disparity the parent note insists on. A positive gap means group $a$'s innocent members are more often wrongly flagged — group $a$ is harmed.
2. Flipped. $\Delta =0.25-0.45=-0.20$. Why this step? To show the metric handles both signs — the sign tells you which group suffers, the magnitude tells you how much.
3. Magnitude. Both cases have $|\Delta |=0.20=20$ percentage points of disparity, just pointing opposite ways. Why this step? Reporting the absolute size separately from the sign lets an auditor rank how severe a disparity is independently of which group it hurts — severity and direction are different questions.

Verify: A perfectly fair model (equal FPRs) would give $\Delta =0$. Here $|\Delta |=0.20\ne 0$, so both scenarios flag a real problem; only the direction of harm differs. Units: probabilities, dimensionless, in $[-1,1]$ — and both $\pm 0.20$ lie safely inside. ✓

---

Example 4 — Cell D: the zero / degenerate case (parity, no false alarm)

Forecast: Will the metric wrongly scream "bias"?

1. Equal rates. $\Delta =0.30-0.30=0$. Why this step? A good disparity metric must return exactly $0$ when the two conditional rates match — otherwise it invents bias where none exists.
2. Degenerate: empty group. If group $b$ has $0$ innocent defendants, then ${\text{FPR}}_b=P(\hat{y}=1\mid y=0,b)$ is $\frac{0}{0}$ — undefined, not zero. Why this step? You cannot divide by zero. The honest answer is "insufficient data", never a fabricated $\Delta$. Reporting a number here would be a lie.
3. What to do. Flag the sample as too small / collect more data before claiming fairness or unfairness. Why this step? An audit that reports a fabricated "0 = fair" on an empty group would let a genuinely biased system pass unnoticed — so refusing to output a number is the safe action, not a cop-out.

Verify: $\Delta =0$ in the equal-rate case is the fixed point of the formula (subtracting equal numbers). The degenerate case correctly yields "undefined", matching real-world audit practice. ✓

---

Example 5 — Cell E: the limiting case (accuracy → 1, stakes → ∞)

Forecast: $99.9\%$ sounds nearly perfect — is it?

1. Failures per year. Failure rate $=1-0.999=0.001$. $\text{Failures}=0.001\times 100,000=100.$ Why this step? Accuracy hides absolute counts. The limit accuracy $\to 1$ still leaves failures $=(1-\text{acc})\times N$, which is large when $N$ is large.
2. Interpret the limit. As accuracy $\to 1$, failures $\to 0$ only if $N$ stays fixed. With $100$ mystery failures no one can explain, you cannot debug, cannot certify, cannot assign liability. Why this step? This is the parent's "safety" pillar quantified — even a near-perfect black box produces failures you must be able to explain.

Verify: Cross-check: if accuracy were exactly $1.000$, failures $=0\times 100,000=0$, consistent with the formula. The gap between "$99.9\%$ feels perfect" and "$100$ real disasters" is the whole point. ✓

---

Example 6 — Cell F: local vs global mismatch

Forecast: Does one clean local explanation clear the model?

1. Distinguish the two scopes. Local interpretability answers "why this one prediction?"; global answers "how does the model behave overall?" (parent's two Types). Why this step? A clean local story can coexist with a globally biased model — they are different questions.
2. Weigh the evidence. ZIP code at $0.40$ global importance (the largest of all features, since $0.40>0.15$ and larger than any single share inside the remaining $0.45$) is a classic proxy for race/income geography (see Example 8). Why this step? The debt-to-income explanation for one person does not neutralise a model that leans on ZIP for everyone.
3. Conclusion. Not trustworthy without a global fairness audit — the local explanation is necessary but not sufficient. Why this step? Certifying a model on a single flattering local example would let a globally ZIP-driven (hence proxy-biased) model into production — so we must escalate to a global audit before signing off.

Verify: Global feature importances sum to $1$: $0.40+0.15+0.45=1.00$, a consistent decomposition, and ZIP at $0.40$ genuinely dominates debt-to-income at $0.15$. ✓

---

Example 7 — Cell G: real-world word problem (adverse-action reasons)

Forecast: Which four make the legal cut?

1. Sort descending. $0.35,0.25,0.20,0.10,0.06,0.04$. Why this step? "Principal reasons" means the largest contributors; we must rank before we can pick the top four.
2. Take the top 4: late payments, credit utilization, account age, hard inquiries. ${\text{sum}}_{\text{top4}}=0.35+0.25+0.20+0.10=\mathrm{0.90.}$ Why this step? We show the notice is honest: the four disclosed reasons explain $90\%$ of the decision, so we are not hiding the real driver.
3. Check the tail. Remaining features sum to $0.06+0.04=0.10$ — minor, legitimately omitted. Why this step? Confirming the undisclosed features are individually and collectively tiny proves the notice is not concealing a large hidden driver — a legal requirement, not a courtesy.

Verify: All contributions sum to $0.35+0.25+0.20+0.10+0.06+0.04=1.00$, a valid decomposition, and top-4 share $0.90\ge 0.5$ so it is genuinely the majority. See Model Cards and Documentation for how such disclosures get recorded. ✓

---

Example 8 — Cell H: exam-style twist (the hidden proxy)

Forecast: Does deleting the sensitive column fix it?

1. Recall the disparity condition. Using $P(\cdot )$ from the definition box, bias exists when $P(\hat{y}=1\mid \text{features},\text{race}=a)\ne P(\hat{y}=1\mid \text{features},\text{race}=b).$ Why this step? The definition conditions on race regardless of whether race is a column; a proxy that carries race information re-introduces exactly this dependence, so the inequality can still hold.
2. Quantify the leaked signal with $r^2$. Shared variance between two variables is $r^2$. Here $r^2=0.85^2=\mathrm{0.7225.}$ Why this step? We just defined $r^2$ as the fraction of one variable's variation explained by the other. So $r^2\approx 0.72$ means about $72\%$ of the variation in race is still recoverable from ZIP alone — the model can reconstruct most of the sensitive attribute it "lost".
3. Connect back to fairness. Because ZIP carries $\approx 72\%$ of the race signal, conditioning on ZIP is almost conditioning on race. The two conditional approval rates in Step 1 therefore still differ, so $\Delta$ (from Example 3) stays non-zero: the disparity survives. Why this step? This closes the loop — deleting the column changed the name of the input but not the information, and the fairness metric still fires.
4. Conclusion. Roughly $72\%$ of the racial signal survives inside ZIP, so the model remains biased. This is why interpretability tools must scan features for proxy correlations with protected attributes, not merely confirm the sensitive column was dropped. Why this step? Stating the operational lesson turns the number into an audit rule: check proxies, because a passed "column removed" test does not clear the model.

Verify: $r^2=0.85^2=0.7225$, and since $0\le r^2\le 1$ this is a valid variance fraction; $0.7225>0.5$ confirms "majority of the signal retained", so the disparity in Step 3 cannot be cleared by dropping race alone. See Fairness in Machine Learning and Adversarial Examples for related proxy attacks. ✓

---

Recall Self-test (reveal after trying)

In Example 1, which model won and by how much? ::: Model Y (interpretable), $0.79$ vs $0.52$ — a margin of $0.27$. In Example 5, how many unexplained failures per year at $99.9\%$ accuracy over $100,000$ ops? ::: $100$. In Example 8, what fraction of the racial signal does ZIP retain if $r=0.85$? ::: $r^2=0.7225\approx 72\%$. Why is $\Delta =0$ the "no bias" fixed point in Example 4? ::: Because subtracting two equal FPRs gives zero, so a fair model lands exactly on it.

See also: Model Transparency vs Model Accuracy · Inherently Interpretable Models · Explainable AI (XAI) Techniques