Concept-based explanations

What Are Concept-based Explanations?

The shift:

• Feature attribution → "Pixel (x, y) contributed +0.8 to the output."
• Concept-based → "The presence of 'stripes' contributed +2.3 to classifying this as a zebra."

How It Works: Deriving TCAV (Testing with Concept Activation Vectors)

TCAV (Kim et al., 2018) is the canonical concept-based method. Let's build it from scratch.

Step 1: What is a Concept Activation Vector (CAV)?

Suppose we want to test if a model uses the concept "striped."

1. Gather concept examples: Collect images with stripes (P) and random images without stripes (N).
2. Extract activations: Feed both sets through the model, extract activations at layer $\ell$: $\mathbf{h}_\ell \in \mathbb{R}^d$.
3. Train a linear classifier: Learn a hyperplane separating P from N in activation space:

where $y_i = +1$ for concept examples, $-1$ for random. The normal vector $\mathbf{v}_c$ (after normalization) is the CAV — it points in the "striped" direction in activation space.

Why this works: If the model represents "stripes" internally, striped images cluster in activation space. The CAV captures that clustering direction.

Step 2: Measuring Concept Sensitivity

Now we test: Does the model's prediction for class $k$ depend on concept $c$?

Define the directional derivative of the logit $S_k(\mathbf{x})$ for class $k$ w.r.t. the concept:

Breaking it down:

• $\nabla_{\mathbf{h}_\ell} S_k(\mathbf{x})$: gradient of class $k$'s logit w.r.t. layer $\ell$ activations. This tells us "how to change $\mathbf{h}_\ell$ to increase $S_k$."
• $\nabla_{\mathbf{h}_\ell} S_k \cdot \mathbf{v}_c$: dot product. If positive, moving $\mathbf{h}_\ell$ toward the concept direction increases $S_k$.
• $\mathbb{1}\{\cdot > 0\}$: binary indicator. We count the fraction of examples where the concept positively influences the class.

Why the expectation? We test on multiple examples of class $k$ (e.g., many zebra images) to avoid noise from a single sample.

Step 3: Statistical Significance

Run TCAV with multiple random concept sets (same size as P, drawn from N). If the real CAV's score is significantly higher than random, the concept is genuinely used.

Typical threshold: $p < 0.05$.

Worked Example: "Striped" for Zebra Classification

Setup: ImageNet classifier, class $k$ = zebra, concept $c$ = "striped."

1. Collect concept examples: 50 images with stripes (tigers, barcode patterns, striped shirts), 50 random ImageNet images.
2. Extract activations: Use layer mixed5c (Inception V3). Each image → $\mathbf{h} \in \mathbb{R}^{2048}$.
3. Train CAV: Logistic regression on the 100 activations. Get $\mathbf{v}_{\text{striped}} \in \mathbb{R}^{2048}$.
4. Test on zebras: Take 100 zebra images. For each:
   • Compute $\nabla_{\mathbf{h}} S_{\text{zebra}}(\mathbf{x})$ via backprop.
   • Check if $\nabla_{\mathbf{h}} S_{\text{zebra}} \cdot \mathbf{v}_{\text{striped}} > 0$.
5. Result: 87% of zebras have positive directional derivative → $\text{TCAV} = 0.87$.
6. Compare to random: Run 20 random CAVs → mean TCAV = 0.52, std = 0.08. The score 0.87 is $\frac{0.87 - 0.52}{0.08} \approx 4.4\sigma$ away → $p < 0.001$. Conclusion: The model genuinely uses "stripes" to classify zebras.

Why this step? Without statistical testing, a TCAV of 0.6 could be noise. The random baseline controls for spurious correlations in activation space.

ACE: Automated Concept Extraction

How it works:

1. Segment inputs: Use image segmentation (e.g., SLIC superpixels) to extract patches.
2. Cluster activations: Run k-means on layer activations of all patches. Each cluster = a discovered concept.
3. Name concepts: Manually inspect cluster examples or use a captioning model.
4. Run TCAV: Test each discovered concept's importance via TCAV.

Advantage: Fully data-driven. Discovers concepts you didn't think to test (e.g., "model uses background grass to classify cows").

Limitation: Cluster quality depends on segmentation. Semantic meaning isn't guaranteed.

Common Mistakes

Connections

• Feature Attribution Methods — Concept-based explanations complement pixel-level saliency by adding semantic meaning.
• Layer-wise Relevance Propagation — LRP traces importance to inputs; TCAV traces importance to concepts.
• Adversarial Examples — Concept-based methods can reveal spurious concepts the model exploits (e.g., "watermark" in medical images).
• Model Editing — Once you identify harmful concepts (via TCAV), you can ablate the corresponding CAV directions.
• Probing Classifiers — Both probe representation spaces, but probing asks "is concept X encoded?" while TCAV asks "is X used for decision Y?"

Key Takeaways

1. Concept-based explanations translate model internals into human semantics.
2. TCAV measures concept importance via directional derivatives of class logits.
3. Statistical testing (random CAVs) is mandatory to avoid false positives.
4. ACE automates concept discovery via clustering, but trades control for coverage.
5. Concepts must be atomic (single semantic) and causally tested (not just present).

Flashcards

#flashcards/ai-ml

Concept Map

Hinglish (regional understanding)