Question bank — Importance of interpretability

False — accuracy measures how often it is right on the test data, not why. It could be right by exploiting a spurious cue (like the snow-vs-wolf background) that vanishes in the real world, so doctors cannot verify its reasoning matches medicine.

False — accuracy asks "how often correct?", interpretability asks "can a human follow the reasoning?". A random forest can be very accurate and nearly impossible to read; a linear model can be perfectly readable and wrong. See Model Transparency vs Model Accuracy.

False in practice — a tree with thousands of nodes and depth 40 is technically transparent but no human can hold it in their head. Interpretability depends on human comprehension, so size and complexity matter even for "inherently interpretable" models (Inherently Interpretable Models).

False — global explains the whole model's behaviour across all inputs ("which features matter overall"), while local explains one prediction ("why this patient got 78%"). A model can be locally explainable per-case yet globally incomprehensible.

False — post-hoc methods (Post-hoc Interpretability Methods) build an approximation or a story about the model; the explanation can be faithful or misleading, and two methods may disagree on the same prediction. Treat them as evidence, not ground truth.

False — Article 22 guarantees safeguards (human intervention, express your view, contest the decision) for qualifying decisions; the strict "right to explanation" is derived from non-binding recitals and remains legally debated. Interpretability is still a practical necessity to offer those safeguards.

False — bias enters through proxies: zip code, prior-arrest counts, and other features correlate with protected attributes. Dropping the sensitive column does not remove the historical discrimination baked into correlated features (Fairness in Machine Learning).

Mostly true but not guaranteed — a linear model with 5,000 features, heavy interactions, and unintuitive units can be harder to read than a shallow 3-leaf tree. "Simple" must mean cognitively simple to the intended human, not just fewer parameters.

False — even unregulated teams need it to debug (find the snow shortcut), to build trust with users, and to catch safety failures. Regulation is one of five pillars, not the only one.

The error is treating interpretability as optional paperwork. In high-stakes or regulated settings you must be able to contest and audit decisions from day one; retrofitting post-hoc explanations onto a live black box may reveal it was learning the wrong thing all along — after harm is done.

High test accuracy does not certify which features were used. The model latched onto background snow — a spurious correlation present in both train and test sets — so both scores looked great while the logic was broken. Only interpretability (saliency maps) exposed it.

Equal overall accuracy hides how errors distribute. COMPAS had similar overall accuracy but a higher false-positive rate for Black defendants. Fairness requires comparing the two groups' conditional distributions, not a single aggregate number.

Wrong comparison baseline — disparity is measured by comparing the two group-conditional rates against each other, $P(\hat{y}=1\mid \text{feat},a)$ vs $P(\hat{y}=1\mid \text{feat},b)$. Comparing a group to the marginal (population) mixes the groups together and can hide or fake a gap.

Attention/saliency shows where the model looked, not that the conclusion is right — it could attend to the correct object yet mislabel it, or the map itself can be unfaithful. Safety validation needs corroboration (radar + lidar agreement), not attention alone.

This drops the real cost terms. Value $=\text{Accuracy}-{\lambda }_1\text{DeploymentRisk}-{\lambda }_2\text{ComplianceCost}-{\lambda }_3\text{TrustGap}$; in banking fraud an 85% interpretable model can beat a 90% black box once those subtracted costs are counted.

Test data can share the same spurious shortcut as training data (snow appears with wolves in both), so the shortcut keeps scoring well. Accuracy measures agreement with labels, never the causal path the model used to get there.

Trust $=f(\text{Accuracy},\text{Interpretability},\text{DomainAlignment})$ — a reasoning a doctor can verify against medical knowledge may be used, while an unexplained higher-accuracy model sits unused. Adoption depends on the whole equation, not accuracy alone.

To provide the guaranteed safeguards — letting a person contest a decision or a regulator audit it — you must be able to inspect and articulate the model's reasoning. You also need it to prove a loan denial wasn't discriminatory (liability).

Metrics like demographic parity tell you that a gap exists but not which feature carries the historical discrimination. Interpretability points to the culprit (e.g. "prior arrests" weighted heavily), which is what lets you actually intervene.

An inherently interpretable model's explanation is its true mechanism, so it cannot mislead; a post-hoc explanation only approximates the black box and can be unfaithful or gamed. When wrong reasoning could kill someone, faithful-by-construction beats plausible-sounding.

If you can see which input regions drive a prediction, you can notice when tiny, meaningless perturbations flip the output — a hallmark of Adversarial Examples. Understanding the decision surface exposes brittleness that raw accuracy hides.

Perfectly interpretable but useless — its "reasoning" is a single constant rule anyone can state, showing that interpretability and usefulness are independent axes. Maximum transparency, minimum predictive value.

Only nominally — each coefficient is readable, but a human cannot survey millions of them to grasp global behaviour. This is the boundary where a "transparent" model class stops being comprehensible in practice.

Possibly neither fully — each is an approximation with its own assumptions, so disagreement signals low faithfulness rather than one being "the truth". Treat contradictory explanations as a warning to distrust the black box, and document it (see Model Cards and Documentation).

Not wasted, but low-priority — the decision framework says low-stakes + pure prediction can favour a black box, adding post-hoc tools only if a bug or complaint appears. The cost weights ${\lambda }_i$ are small here, so accuracy dominates the value equation.

The condition $P(\hat{y}=1\mid \text{feat},a)\ne P(\hat{y}=1\mid \text{feat},b)$ is stated for otherwise comparable features precisely to isolate group effect from feature effect. If groups differ in the very features conditioned on, you must control for that before concluding the model, not the data, is unfair.

Its explanations are unstable, so any single explanation is untrustworthy — an unstable decision surface is itself a red flag (often near adversarial vulnerability). Interpretability here reveals a fragility problem, which is a finding, not a failure of the method.