Concept-based explanations

6.3.11 · AI-ML › Interpretability & Explainability

Concept-based Explanations Hain Kya?

Shift yeh hai:

• Feature attribution → "Pixel (x, y) ne output mein +0.8 contribute kiya."
• Concept-based → "'Stripes' ki presence ne ise zebra classify karne mein +2.3 contribute kiya."

Yeh Kaise Kaam Karta Hai: TCAV Derive Karna (Testing with Concept Activation Vectors)

TCAV (Kim et al., 2018) canonical concept-based method hai. Isse scratch se samjhte hain.

Step 1: Concept Activation Vector (CAV) Kya Hota Hai?

Maano hum test karna chahte hain ki model "striped" concept use karta hai ya nahi.

1. Concept examples collect karo: Stripes wali images (P) aur bina stripes ki random images (N) lo.
2. Activations extract karo: Dono sets ko model se feed karo, layer $\ell$ par activations extract karo: ${\mathbf{h}}_{\ell }\in {\mathbb{R}}^d$.
3. Linear classifier train karo: Activation space mein P aur N ko alag karne wala hyperplane seekho:

jahan $y_i=+1$ concept examples ke liye, $-1$ random ke liye. Normal vector ${\mathbf{v}}_c$ (normalization ke baad) hi CAV hai — yeh activation space mein "striped" direction ki taraf point karta hai.

Yeh kyun kaam karta hai: Agar model "stripes" ko internally represent karta hai, toh striped images activation space mein cluster ho jaati hain. CAV usi clustering direction ko capture karta hai.

Step 2: Concept Sensitivity Measure Karna

Ab hum test karte hain: Kya class $k$ ke liye model ki prediction concept $c$ par depend karti hai?

Class $k$ ke logit $S_k(\mathbf{x})$ ka concept ke saath directional derivative define karo:

Tukda tukda samjho:

• ${\nabla }_{{\mathbf{h}}_{\ell }}{S}_k(\mathbf{x})$: class $k$ ke logit ka layer $\ell$ activations ke w.r.t. gradient. Yeh batata hai "$S_k$ badhane ke liye ${\mathbf{h}}_{\ell }$ ko kaise change karein."
• ${\nabla }_{{\mathbf{h}}_{\ell }}{S}_k\cdot {\mathbf{v}}_c$: dot product. Agar positive hai, toh ${\mathbf{h}}_{\ell }$ ko concept direction ki taraf move karne se $S_k$ badhta hai.
• $1\{\cdot >0\}$: binary indicator. Hum un examples ka fraction count karte hain jahan concept class ko positively influence karta hai.

Expectation kyun? Hum class $k$ ke multiple examples par test karte hain (jaise kai zebra images) taaki ek single sample ka noise avoid ho.

Step 3: Statistical Significance

TCAV ko multiple random concept sets ke saath run karo (same size as P, N se drawn). Agar real CAV ka score random se significantly zyada hai, toh concept genuinely use hota hai.

Typical threshold: $p<0.05$.

Worked Example: Zebra Classification ke liye "Striped"

Setup: ImageNet classifier, class $k$ = zebra, concept $c$ = "striped."

1. Concept examples collect karo: 50 striped images (tigers, barcode patterns, striped shirts), 50 random ImageNet images.
2. Activations extract karo: Layer mixed5c (Inception V3) use karo. Har image → $\mathbf{h}\in {\mathbb{R}}^{2048}$.
3. CAV train karo: 100 activations par Logistic regression. ${\mathbf{v}}_{\text{striped}}\in {\mathbb{R}}^{2048}$ milta hai.
4. Zebras par test karo: 100 zebra images lo. Har ek ke liye:
   • Backprop se ${\nabla }_{\mathbf{h}}{S}_{\text{zebra}}(\mathbf{x})$ compute karo.
   • Check karo ki ${\nabla }_{\mathbf{h}}{S}_{\text{zebra}}\cdot {\mathbf{v}}_{\text{striped}}>0$ hai ya nahi.
5. Result: 87% zebras ka positive directional derivative hai → $\text{TCAV}=0.87$.
6. Random se compare karo: 20 random CAVs run karo → mean TCAV = 0.52, std = 0.08. Score 0.87, $\frac{0.87-0.52}{0.08}\approx 4.4\sigma$ door hai → $p<0.001$. Conclusion: Model genuinely "stripes" use karta hai zebras classify karne ke liye.

Yeh step kyun? Statistical testing ke bina, 0.6 ka TCAV noise ho sakta hai. Random baseline spurious correlations ko activation space mein control karta hai.

ACE: Automated Concept Extraction

Yeh kaise kaam karta hai:

1. Inputs segment karo: Image segmentation (jaise SLIC superpixels) use karke patches extract karo.
2. Activations cluster karo: Sabhi patches ke layer activations par k-means run karo. Har cluster = ek discovered concept.
3. Concepts ko naam do: Cluster examples manually inspect karo ya captioning model use karo.
4. TCAV run karo: TCAV ke zariye har discovered concept ki importance test karo.

Faida: Poora data-driven hai. Aisi concepts discover karta hai jo tumne test karne ke baare mein socha bhi nahi tha (jaise "model background grass use karta hai cows classify karne ke liye").

Limitation: Cluster quality segmentation par depend karti hai. Semantic meaning guaranteed nahi hai.

Common Mistakes

Connections

• Feature Attribution Methods — Concept-based explanations pixel-level saliency ko semantic meaning add karke complement karti hain.
• Layer-wise Relevance Propagation — LRP importance ko inputs tak trace karta hai; TCAV importance ko concepts tak trace karta hai.
• Adversarial Examples — Concept-based methods un spurious concepts ko reveal kar sakti hain jo model exploit karta hai (jaise medical images mein "watermark").
• Model Editing — Ek baar harmful concepts identify ho jayein (TCAV ke zariye), tum corresponding CAV directions ko ablate kar sakte ho.
• Probing Classifiers — Dono representation spaces probe karte hain, lekin probing poochti hai "kya concept X encoded hai?" jabki TCAV poochta hai "kya X decision Y ke liye use hota hai?"

Key Takeaways

1. Concept-based explanations model internals ko human semantics mein translate karti hain.
2. TCAV class logits ke directional derivatives ke zariye concept importance measure karta hai.
3. False positives se bachne ke liye Statistical testing (random CAVs) mandatory hai.
4. ACE clustering ke zariye concept discovery automate karta hai, lekin control ke badle coverage milti hai.
5. Concepts atomic (single semantic) hone chahiye aur causally tested (sirf present nahi) hone chahiye.

Flashcards

#flashcards/ai-ml

Concept Map