Feature attribution (SHAP, LIME) — Hinglish notes

6.3.2 · AI-ML › Interpretability & Explainability

Core concepts

Feature attribution har input feature ko ek single prediction ke liye numerical importance score assign karta hai. Global feature importance (jaise permutation importance) se alag, attribution instance-specific hota hai: wahi feature ek prediction ke liye critical ho sakta hai aur doosre ke liye irrelevant.

Challenge yeh hai: modern models (deep nets, ensembles) non-linear, interactive compositions hain. Kisi feature ka contribution doosre features ki values par depend karta hai. Hume ek principled tarika chahiye $f (x)$ ko additive contributions mein decompose karne ka.

Figure — Feature attribution (SHAP, LIME)

LIME: Local Interpretable Model-agnostic Explanations

Core idea

LIME model ko ek black box maanta hai aur ek locally faithful linear approximation banata hai.

Linear kyun? Humans linear models ko aasani se interpret karte hain: "feature $x_{i}$ prediction ko $w_{i}$ per unit badhata hai." Chahe $f$ globally non-linear ho, $x$ ke paas woh roughly linear ho sakta hai.

Yeh kaise kaam karta hai:

Perturbations generate karo: $x$ ke paas naaye instances $z \in Z$ randomly features toggle/perturb karke sample karo.
Proximity se weigh karo: Weight $π_{x} (z) = exp (- \frac{d ( x , z ) ^{2}}{σ ^{2}})$ assign karo taaki close neighbors zyada matter karein.
Local linear model fit karo: Solve karo

ξ (x) = ar g g \in G min L (f, g, π_{x}) + Ω (g)

jahan $G$ linear models ki class hai, $L$ weighted squared loss hai, $Ω (g)$ complexity ko penalize karta hai (jaise, L1 se kuch features select hote hain).

$g$ ke coefficients LIME explanations hain: $g (z) = w_{0} + \sum_{i = 1}^{d} w_{i} z_{i}$ .

Worked example Image classification ke liye LIME

Task: Explain karo kyun ek CNN ne ek image ko "dog" classify kiya. Step 1 – Image segment karo: Superpixels mein divide karo (jaise, 50 regions).

Step 2 – Perturb karo: Superpixels ko on/off (gray se mask) karke samples banao. Jaise, sample $z_{1}$ mein regions {3, 7, 12 ...} visible hain, baaki masked hain.

Step 3 – Predictions lo: $f (z_{1}) = 0.92$ (dog), $f (z_{2}) = 0.15$ (dog), etc.

Step 4 – Similarity se weight karo: $π_{x} (z) = exp (- \frac{#different pixels}{2 σ ^{2}})$ .

Step 5 – Linear model fit karo: Har superpixel ek binary feature hai (on/off). $g (z) = w_{0} + \sum_{j = 1}^{50} w_{j} z_{j}$ weighted regression se fit karo. Positive $w_{j}$ ka matlab hai superpixel $j$ "dog" score badhata hai.

Result: Top 5 positive-weight superpixels dog ka face highlight karte hain. Yahi explanation hai.

Gray se mask kyun? Hume ek "neutral" baseline chahiye jo model ko confuse na kare. Gray ImageNet mean ke close hota hai.

SHAP: SHapley Additive exPlanations

Game-theoretic foundation

SHAP cooperative game theory ke Shapley values par based hai. Socho features ek coalition mein players hain, aur "payout" model ki prediction hai.

Sawaal: Har player (feature) $f (x)$ aur average prediction $f (\emptyset)$ ke beech ke difference mein kitna contribute karta hai?

Definition Shapley value

Feature $i$ ke liye, Shapley value hai:

ϕ_{i} (f, x) = S \subseteq {1, \dots, d} ∖ {i} \sum \frac{∣ S ∣ ! ( d - ∣ S ∣ - 1 )!}{d !} [f_{S} (x_{S \cup {i}}) - f_{S} (x_{S})]

Yeh kya kehta hai: Saare subsets $S$ consider karo jo $i$ ko include nahi karte. $i$ ko $S$ mein add karo aur marginal gain measure karo. Saari orderings par average karo (factorials combinatorial weights hain).

Yeh weights kyun? Yeh axiom se aate hain ki symmetric roles mein players ko equal credit milta hai. Yeh unique attribution hai jo satisfy karta hai:

Efficiency: $\sum_{i = 1}^{d} ϕ_{i} = f (x) - f (\emptyset)$ (contributions total deviation ke barabar hain).
Symmetry: Agar $i$ aur $j$ saari coalitions mein equally contribute karte hain, toh $ϕ_{i} = ϕ_{j}$ .
Dummy: Agar $i$ kabhi prediction nahi badalta, toh $ϕ_{i} = 0$ .
Linearity: $f = f_{1} + f_{2}$ ke liye, $ϕ_{i} (f) = ϕ_{i} (f_{1}) + ϕ_{i} (f_{2})$ .

Scratch se DERIVATION:

Setup: Hamare paas $d$ features hain. Subset $S$ ke liye, $f_{S} (x_{S})$ define karo expected prediction ke roop mein jab hum $S$ ke features observe karte hain aur baaki ko marginalize karte hain:

f_{S} (x_{S}) = E_{X_{- S}} [f (x_{S}, X_{- S})]

Marginalize kyun? Missing features ko zeros ke bajaye unki typical distribution se replace karte hain.

Marginal contribution: Jab hum feature $i$ ko coalition $S$ mein add karte hain, gain hai: $Δ_{i} (S) = f_{S \cup {i}} (x_{S \cup {i}}) - f_{S} (x_{S})$
Saari orderings par average karo: Features ki $d!$ orderings hoti hain. Har ordering mein, $i$ kuch set $S$ ke baad aata hai. Un orderings ki sankhya jahan $i$ exactly $S$ ke features ke baad aata hai: $∣ S ∣! \cdot (d - ∣ S ∣ - 1)!$ . Toh us configuration ki probability hai:

\frac{∣ S ∣ ! ( d - ∣ S ∣ - 1 )!}{d !}

Saare $S \neq ∋ i$ par $Δ_{i} (S)$ ka weighted average Shapley value deta hai.

Result: $ϕ_{i}$ feature $i$ ko fair credit hai baseline $f (\emptyset)$ se $f (x)$ tak prediction move karne ke liye.

SHAP compute karna: challenge

Exact Shapley values ke liye $2^{d}$ evaluations chahiye (saare subsets). High-dimensional data ke liye yeh intractable hai.

Solutions:

KernelSHAP (model-agnostic): Shapley values ko ek weighted linear model fit karke approximate karta hai (LIME jaisa, lekin specific Shapley weights ke saath). Samples →∞ hone par true Shapley par converge karta hai.
TreeSHAP (tree models ke liye): Tree structure exploit karta hai exact Shapley values polynomial time mein compute karne ke liye. Kaise: Tree traverse karo, feature splits track karo, aur paths par expectations compute karo.
DepSHAP (neural nets ke liye): Backpropagation + reference values use karke approximate karta hai.

Worked example Loan default ke liye SHAP

Scenario: Model predict karta hai P(default) = 0.72 applicant Bob ke liye. Base rate $ϕ_{0} = 0.20$ .

Features:

Credit score: 580
Income: $45k
Debt-to-income: 0.45
Age: 28 Step 1 – Coalitions sample karo: Subset $S = {credit, income}$ ke liye, $f_{S}$ compute karo bahut saare random samples par average karke jahan debt aur age unki distribution values se replace hain, lekin credit=580 aur income=45k fixed hain.

Step 2 – Marginal gains:

Credit ko $\emptyset$ mein add karo: $f_{{cr e d i t}} - f_{\emptyset} = 0.35 - 0.20 = + 0.15$
Income ko ${cr e d i t}$ mein add karo: $f_{{cr e d i t, in co m e}} - f_{{cr e d i t}} = 0.40 - 0.35 = + 0.05$
Debt ko ${cr e d i t, in co m e}$ mein add karo: $f_{{cr e d i t, in co m e, d e b t}} - f_{{cr e d i t, in co m e}} = 0.58 - 0.40 = + 0.18$
Age ko ${cr e d i t, in co m e, d e b t}$ mein add karo: $f_{all} - f_{{cr e d i t, in co m e, d e b t}} = 0.72 - 0.58 = + 0.14$

Step 3 – Orderings par average karo: Saari $4! = 24$ orderings ke liye repeat karo. Shapley value $ϕ_{debt}$ saari orderings mein debt ka average marginal contribution hai.

Result (exact computation ke baad):

$ϕ_{credit} = + 0.12$ (low score default risk badhata hai)
$ϕ_{income} = - 0.05$ (moderate income thoda protective hai)
$ϕ_{debt} = + 0.31$ (high debt ratio sabse bada driver hai)
$ϕ_{age} = + 0.14$ (young age risk add karta hai)

Check karo: $0.20 + 0.12 - 0.05 + 0.31 + 0.14 = 0.72$ . ✓

Yeh ordering-average kyun? Kyunki features interact karte hain. Agar hum jaante hain credit low hai, toh debt add karne ka effect alag hoga agar credit high hota. Orderings par average karna saare interaction contexts account karta hai.

Worked example TreeSHAP in action

Task: 3 features (A, B, C) wala random forest house price predict karta hai. Ek tree split karta hai: A > 5 → B > 3 → leaf₁( $250 k), nahi t o h l e a f_{2} ($ 180k).

Instance: A=6, B=4, C=1. Prediction = $250 k . B a se =$ 200k.

TreeSHAP algorithm:

Path trace karo: root → (A > 5: yes) → (B > 3: yes) → leaf₁.
Har split par compute karo: agar feature unknown hota, toh kitna fraction training data left vs right jaata? Woh fraction alternative branch weight karta hai.
Feature A: split right kiya ( $250 k k i t a r a f) . A g a r A u nk n o w nh o t a,$ 60% $d a t a l e f t ($ 180k) jaata, $40%$ right ( $250 k) . A k e bina e x p ec t e d :$ 0.6 \cdot 180 + 0.4 \cdot 250 = 208 $. A k ama r g ina l co n t r ib u t i o n =$ 250 - 208 = +42$.
Feature B: A > 5 branch mein, higher leaf ki taraf split kiya. A > 5 branch mein B ke bina expected: $220$ . Marginal = $250 - 220 = + 30$ .
Feature C: kabhi split nahi kiya. $ϕ_{C} = 0$ . Sum: $ϕ_{A} + ϕ_{B} + ϕ_{C} = 42 + 30 + 0 = 72$ . Lekin wait, $250 - 200 = 50$ . Error? Nahi: TreeSHAP training data fractions ko integration measure ke roop mein use karta hai. Yahan numbers illustrative hain; exact TreeSHAP tree ki coverage statistics ke through saari coalitions account karta hai.

LIME vs SHAP: tradeoffs

Aspect	LIME	SHAP
Foundation	Locally faithful linear model	Shapley values (game theory)
Guarantees	Koi nahi (approximation-dependent)	Efficiency, symmetry, dummy axioms
Compute cost	$O (n_{samples})$ , flexible	$O (2^{d})$ exact; TreeSHAP $O (T L D^{2})$ , KernelSHAP $O (n \cdot 2^{d})$ approx
Stability	Kam samples ke saath high variance	Zyada stable (principled)
Additivity	Koi guarantee nahi $\sum w_{i} \approx f (x) - base$	Exact: $\sum ϕ_{i} = f (x) - ϕ_{0}$
Use case	Quick, interpretable, jab consistency critical nahi	Jab theoretically sound, consistent explanations chahiye

LIME kyun use karein? Speed + simplicity. Tum ek one-off prediction ko reasonable samples ke saath seconds mein explain kar sakte ho. Prototyping aur user-facing apps ke liye badhiya jahan "roughly right" kaafi ho.

SHAP kyun use karein? Jab decisions high-stakes hoon (healthcare, finance) aur tumhe defensible, consistent explanations chahiye. SHAP ki additivity ka matlab hai tum ek prediction ko exactly decompose kar sakte ho. TreeSHAP tree models ke liye fast bhi hai.

Recall Ek 12-saal ke bachche ko explain karo

Socho tumhara teacher tumhe ek group project par grade deta hai. Tum jaanna chahte ho: "Maine kitna contribute kiya vs mere teammates ne?"

LIME aisa hai jaise paas ke kuch classmates se poochho: "Agar main group mein nahi hota, toh kya grade kam hota?" Phir unke jawaabon ka average lo. Yeh quick hai, lekin agar alag logon se poochho, toh alag jawab mil sakte hain.

SHAP aisa hai jaise teacher carefully calculate kare: "Agar main imagine karun saare possible groups with aur without tumhare, aur average karun ki grade kitna badlta hai jab tum add hote ho, wahi tumhara exact contribution hai." Yeh fair hai, sab isse agree karte hain, lekin compute karne mein zyada time lagta hai.

Dono tumhe samajhne mein help karte hain kyun grade (prediction) woh hai jo hai, ek feature (person) ek baar mein.

Connections

6.301-Global-vs-local-interpretability – Feature attribution canonical local method hai.

6.2.03-Attention-mechanisms – Attention weights feature attribution ka ek form hain (halanki Shapley-based nahi).
5.4.02-Permutation-importance – Global feature importance; SHAP ko aggregate karke similar rankings recover ki ja sakti hain.
6.3.03-Counterfactual-explanations – Alternative local explanation: " $x_{i}$ ko $x_{i}^{'}$ mein badlo prediction flip karne ke liye."
3.5.04-Tree-ensembleinterpretation – TreeSHAP fast exact computation ke liye tree structure leverage karta hai.

#flashcards/ai-ml

Feature attribution kya hai? :: Har input feature ko ek single prediction ke liye numerical importance score assign karna, yeh dikhate hue ki har feature ne us specific output mein kitna contribute kiya.

LIME "local" kyun hai?

LIME instance

x

ke aas-paas ek chote neighborhood mein ek simple interpretable model (linear) fit karta hai, proximity

π_{x} (z) = exp (- d (x, z)^{2} / σ^{2})

se weighted.

LIME objective function derive karo :: Weighted squared loss se shuru karo $L = \sum_{z} π_{x} (z) [f (z) - g (z)]^{2}$ ( $x$ ke paas $f$ ke closeness ke liye), complexity penalty $Ω (g) = λ ∥ w ∥_{1}$ (sparsity) add karo, unka sum minimize karo locally faithful sparse linear $g$ paane ke liye.

Shapley values kya hain?

Unique fair attribution jo efficiency (

\sum ϕ_{i} = f (x) - f (\emptyset)

), symmetry, dummy, aur linearity satisfy karta hai. Feature

i

ke liye:

ϕ_{i} = \sum_{S} \frac{∣ S ∣ ! ( d - ∣ S ∣ - 1 )!}{d !} [f_{S \cup {i}} - f_{S}]

, saari coalitions par averaged marginal contribution.

SHAP values prediction mein kyun sum hote hain?

Shapley values ke efficiency axiom se:

f (x) = ϕ_{0} + \sum_{i = 1}^{d} ϕ_{i} (x)

, jahan

ϕ_{0}

base rate hai. Yeh exact hai, approximate nahi.

TreeSHAP polynomial time kaise achieve karta hai?

Tree structure exploit karta hai: har split ke liye, training data ka fraction compute karo jo har direction jaata hai, use alternative branch ki expectation weight karne ke liye use karo. Saare

2^{d}

subsets explicitly enumerate karne ki zaroorat nahi.

LIME high variance mistake aur fix

Bahut kam perturbations use karna (jaise 10) linear fit ko unstable banata hai; explanations ek run se doosre mein wildly vary karte hain. Fix: 1,000–10,000 samples use karo aur multiple runs mein stability check karo.

Key difference: LIME vs SHAP guarantees :: LIME ek approximation hai jisme fidelity ya additivity ki koi theoretical guarantee nahi. SHAP mein axiom-based guarantees hain: efficiency, symmetry, dummy player, jo ise theoretically sound aur consistent banate hain.

SHAP pe LIME kab use karein?

Jab prototyping ya low-stakes user-facing explanations ke liye speed aur simplicity chahiye, aur approximate local fidelity kaafi ho. LIME flexible sampling ke saath kisi bhi model type ko explain kar sakta hai.

Shapley derivation mein

f_{S} (x_{S})

kya hai?

Expected prediction jab subset

S

ke features

x_{S}

par observe kiye jaate hain aur

S

mein nahi hone wale features marginalize kiye jaate hain:

f_{S} (x_{S}) = E_{X_{- S}} [f (x_{S}, X_{- S})]

. Yeh marginal contributions compute karne ke liye baseline define karta hai.