Importance of interpretability: Visual walkthrough

This is a visual walkthrough of the central result in the parent topic. Read it top to bottom; every symbol is built before it is used.

Step 1 — What does "accuracy" even draw?

WHAT. Before we can trade accuracy against anything, we need a picture of what accuracy is.

WHY. People say "90% accurate" as if it were one number floating in the air. But a percentage is really a count of correct answers out of a whole. To weigh it against costs later, we must see it as an area — a slice of a bar — not a floating number.

PICTURE. Look at the figure. We have 100 decisions the model made (the full bar). The blue slice is the fraction it got right. "Accuracy" is just: how much of the bar is blue?

$blue fraction Accuracy = \frac{correct decisions blue count}{full bar all decisions}$

correct decisions — how many times the model matched reality.
all decisions — the total number of predictions (the whole bar).

Figure s01 — A horizontal bar of 100 decisions: a large blue slice (correct, 0.90 of the bar) and a small grey remainder. Accuracy is read off as the blue length divided by the whole bar length.

Step 2 — The hidden cost of the "wrong" slice

WHAT. We now colour the other part of the bar — the mistakes — and give it a price.

WHY. Accuracy quietly assumes every mistake is equally cheap. That is the lie. A mistake that blocks a real customer's honest payment is not the same as a mistake in a photo-tagging app. To compare models fairly, each mistake must carry a weight, a number saying "this is how much it hurts."

PICTURE. The orange slice is the wrong decisions. Above it we stack a red bar whose height is the pain-per-mistake. Cost is area = how many mistakes × how much each hurts.

$Raw Cost = orange fraction (mistakes) (1 - Accuracy) \times pain per mistake c$

$1 - Accuracy$ — the fraction that is not blue, i.e. the orange mistakes (a number in $[0, 1]$ ).
$c$ — the pain of one mistake, itself normalized to $[0, 1]$ (0 = harmless, 1 = worst harm we track) so that Raw Cost stays a unit-less number on the same ruler as accuracy.

The symbol $\times$ is here because "total damage" is always count of bad events times damage each — that is what multiplication answers: repeated addition of the same cost.

Figure s02 — The same bar, now with the wrong decisions coloured orange, and a red "pain per mistake" bar of height $c$ stacked above them. Raw Cost is the shaded area = orange fraction $\times$ height $c$ .

Step 3 — Splitting the pain into named channels

WHAT. We split that single "pain per mistake" into the three named costs the parent named: Deployment Risk, Compliance Cost, Trust Gap. Then we fold the mistake-fraction into each channel so the picture and the algebra agree.

WHY. In the real world, a mistake hurts in distinct ways at once: it can cause an operational failure (deployment risk), trigger a legal/audit expense (compliance cost), and erode users' willingness to use the system (trust gap). Lumping them into one $c$ hides which one dominates in a given domain. We break the red bar into three coloured segments so we can tune each independently.

PICTURE. From Step 2 the total cost was $Raw Cost = (1 - A) \times c$ . First we split the per-mistake pain $c$ into three per-mistake pieces:

$pain per mistake c = deployment λ_{1} r + compliance λ_{2} γ + trust gap λ_{3} g$

$r, γ, g$ — the per-mistake pain in each channel (deployment, compliance, trust), each in $[0, 1]$ .
$λ_{1}, λ_{2}, λ_{3}$ — the domain dials weighting each channel.

Now multiply the whole thing by the mistake fraction $(1 - A)$ , exactly as Step 2 demands — the multiplier does not vanish, it gets absorbed into each channel:

$Raw Cost = (1 - A) c = λ_{1} = R (1 - A) r + λ_{2} = C (1 - A) γ + λ_{3} = G (1 - A) g$

We define the three total channel penalties to already include the mistake fraction:

$R = (1 - A) r, C = (1 - A) γ, G = (1 - A) g$

$R$ = Deployment Risk — the total deployment pain across all mistakes (already $\times (1 - A)$ ). A hidden bug like the snow-detecting husky classifier blowing up in production. Normalized to $[0, 1]$ .
$C$ = Compliance Cost — the total legal/audit pain across all mistakes (see Model Cards and Documentation). Normalized to $[0, 1]$ .
$G$ = Trust Gap — the total adoption loss across all mistakes. Normalized to $[0, 1]$ .

So the " $(1 - A)$ " multiplier never disappears — it lives inside $R$ , $C$ , $G$ . With this bookkeeping the total penalty is simply

$Total Penalty = deployment λ_{1} R + compliance λ_{2} C + trust gap λ_{3} G$

and this is the whole cost across all mistakes (not per-mistake), so it can be subtracted from accuracy directly in Step 4.

Figure s03 — The single red pain bar splits into three stacked segments (red = deployment $λ_{1} R$ , orange = compliance $λ_{2} C$ , grey = trust $λ_{3} G$ ). Each segment's height already has the mistake fraction baked in, so their sum is the total cost.

Step 4 — Where interpretability enters the picture

WHAT. We connect interpretability to those three penalties. This is the key move.

WHY. So far nothing mentioned interpretability. But look at what each penalty depends on: you can only shrink Deployment Risk if you can see what the model learned; you can only pay less Compliance Cost if you can explain decisions to auditors; the Trust Gap shrinks only when humans can check the reasoning. All three shrink as understandability rises. That is precisely why the parent note calls interpretability essential.

PICTURE. Two knobs on the left: Accuracy and Interpretability. Arrows show Accuracy feeds the blue "reward" bar, while Interpretability pulls down on all three penalty segments. More understandability = shorter red bars. Because $R$ , $C$ , $G$ already carry the mistake fraction from Step 3, the subtraction below is clean — no stray $(1 - A)$ factor is needed:

$Value = accuracy, reward A - ↓ as I ↑ λ_{1} R (I) - ↓ as I ↑ λ_{2} C (I) - ↓ as I ↑ λ_{3} G (I)$

$A$ — accuracy from Step 1 (the blue reward), in $[0, 1]$ .
$I$ — interpretability: a unit-less number from $0$ (opaque) to $1$ (fully transparent), the transparency axis.
$R (I), C (I), G (I)$ — each total penalty (mistake fraction already included) is now also a function of $I$ : as $I$ goes up, each goes down.

We write the penalties as functions of $I$ because that is exactly the question we care about: "if I make the model more understandable, what happens to my costs?" A function is the tool that turns "depends on" into arithmetic.

Formula The shape we require of the penalty curves (monotone decreasing)

We do not need the exact formula for $R (I)$ , $C (I)$ , $G (I)$ — only their shape: each must be non-increasing in $I$ , i.e. formally $R^{'} (I) \leq 0, C^{'} (I) \leq 0, G^{'} (I) \leq 0 for all I \in [0, 1] .$

$R^{'} (I)$ ("R-prime of I") — the derivative of $R$ with respect to $I$ : it denotes the instantaneous rate of change of the penalty $R$ as interpretability $I$ nudges up a hair. Picture the slope (steepness) of the $R (I)$ curve at a point — how fast the red bar shrinks as you turn the interpretability knob. A value $\leq 0$ means the curve never tilts upward.
$C^{'} (I)$ and $G^{'} (I)$ — the same idea for the compliance and trust curves: their slopes, both required to point flat-or-downhill.

In words: making a model more interpretable can only lower (or leave unchanged) each penalty, never raise it. The straight downward lines in the figure below are just the simplest curves obeying this rule.

Figure s04 — Three downward lines plot $R (I)$ , $C (I)$ , $G (I)$ against interpretability $I$ (0 opaque → 1 transparent). All slope downhill (slope $\leq 0$ ), so higher $I$ can only shrink each penalty.

Step 5 — The crossover: 85% beats 90%

WHAT. We finally put two real models on the same axes and find where the interpretable one wins.

WHY. This is the parent's shocking claim, made concrete. We take the banking-fraud example: an interpretable model ( $A = 0.85$ , high $I$ ) versus a black box ( $A = 0.90$ , near-zero $I$ ), and compute each one's Value.

PICTURE. Two full bars side by side. Blue = accuracy reward. Red = total penalty. The black box has a taller blue but a much taller red. The interpretable model has shorter blue but tiny red. Net height (Value) — the black arrow — is taller for the interpretable model.

Plug the parent's regulated-banking dials, say $λ_{1} = λ_{2} = λ_{3} = 1$ with an opaque model incurring $R = C = G = 0.10$ each, and an interpretable model incurring near $0$ :

Value_{black box} = 0.90 - (0.10 + 0.10 + 0.10) = 0.60

Value_{interpretable} = 0.85 - (0.00 + 0.00 + 0.00) = 0.85

$0.90$ / $0.85$ — the two accuracies (blue rewards).
$0.30$ vs $0.00$ — the two penalty totals (red bars).
$0.60 < 0.85$ — the interpretable model wins by $0.25$ of Value.

Figure s05 — Two models as bars: blue reward up, red penalty down. Black box (0.90 blue, 0.30 red) nets Value 0.60; interpretable (0.85 blue, 0.00 red) nets Value 0.85. The interpretable bar's net height is taller.

Step 6 — The degenerate cases (never leave a gap)

WHAT. We check the edges: what happens when the dials are extreme.

WHY. A formula you trust must survive its corner cases. If it gives silly answers when $λ = 0$ or when accuracies tie, it was never really understood.

PICTURE. Four mini-panels, one per corner.

All $λ = 0$ (low-stakes photo app). Penalties vanish: $Value = A$ . Formula collapses back to plain accuracy — so the black box's extra $0.05$ does win. Correct: low stakes, use the black box + post-hoc explanation.
Huge $λ_{2}$ (heavily regulated). One giant red segment. Even a tiny compliance gap makes any opaque model lose. Matches GDPR/FCRA pressure toward interpretability.
Equal accuracy ( $A_{b} = A_{i}$ ). Blue bars tie, so whoever has the shorter red bar wins — interpretability decides. This is why, all else equal, prefer an inherently interpretable model.
Interpretable model also more accurate. No trade-off at all — it dominates on both bars. The "trade-off" was never a law, only a common case.

Figure s06 — Four small bar-pair panels: (1) all $λ = 0$ so only blue matters and the box wins; (2) huge $λ_{2}$ so a towering red penalty sinks the box; (3) equal blue bars so the shorter red bar (interpretable) wins; (4) interpretable model taller in blue and shorter in red, dominating outright.

The one-picture summary

WHAT. One figure compresses all six steps: knobs (Accuracy, Interpretability) → reward bar minus three shrinking penalty bars → net Value → crossover point.

WHY. If you remember one image from this page, remember this: Value is a tug-of-war between one blue reward and three red penalties, and interpretability is the hand pulling the red bars down.

Figure s07 — The full story in one bar: blue reward $A$ on top, three red penalty segments ( $λ_{1} R$ , $λ_{2} C$ , $λ_{3} G$ ) subtracted below, a green arrow marking that interpretability $I$ shrinks the red segments, and the net Value (0.60) labelled.

$Value = blue reward A - [three red penalties, shrunk by I λ_{1} R (I) + λ_{2} C (I) + λ_{3} G (I)]$

Recall Feynman retelling — say it like you'd tell a friend

Start with a bar: the blue part is how often the model is right. Everybody stops here and says "bigger blue wins." But the leftover part — the mistakes — costs money, and how much depends on where you use the model. So paint the mistakes red. That red is "how many mistakes times how much each hurts." Now split that pain into three stripes: one for stuff breaking in production, one for lawyers and auditors, one for people refusing to trust it — and keep the "how many mistakes" factor tucked inside each stripe so the bookkeeping stays honest. First we agree everything is measured on the same 0-to-1 ruler, so subtracting is fair. Now here's the trick: making the model understandable is like a hand that reaches over and squishes all three red stripes shorter (never taller — that's the rule, "slope points downhill"). So the real score isn't the blue bar — it's blue minus the whole red stack. A slightly shorter blue bar (85% instead of 90%) can still win if its red stripes are tiny. Turn the "how much do I care" dials to zero (a meme app) and only blue matters — use the black box. Turn any one of them way up (a hospital, a bank, a self-driving car) and that red stripe dominates — you must be able to explain. And sometimes the clear model is even more accurate, so there's no fight at all. That's the whole story: Value = one reward minus three penalties, and interpretability shrinks the penalties.

Recall Quick self-test

What does the blue bar represent? ::: The model's accuracy — the fraction of decisions it got right, a number in $[0, 1]$ . What do the three red segments represent? ::: Deployment Risk $R$ , Compliance Cost $C$ , and Trust Gap $G$ — the three ways a mistake hurts, each with the mistake fraction $(1 - A)$ already baked in, each normalized to $[0, 1]$ . Where did the $(1 - A)$ mistake-fraction multiplier go? ::: It did not vanish — it is folded inside $R = (1 - A) r$ , $C = (1 - A) γ$ , $G = (1 - A) g$ , so the penalties are total costs and can be subtracted from accuracy directly. What does $R^{'} (I)$ mean? ::: The derivative — the instantaneous rate of change (slope) of $R$ as $I$ increases; required to be $\leq 0$ so penalties only shrink. Why is subtracting penalties from accuracy legal? ::: Because every term is put on the same unit-less $[0, 1]$ ruler, so the subtraction is dimensionally consistent. What do subscripts $b$ and $i$ mean? ::: $b$ = the black-box model, $i$ = the interpretable model. Under the parent's banking numbers, who wins and by how much? ::: The interpretable model, $0.85$ vs $0.60$ — a margin of $0.25$ . When does the black box correctly win? ::: When all $λ = 0$ (low stakes), so Value collapses to plain accuracy and its extra $0.05$ decides.