Exercises — Concept-based explanations

(a) concept direction $={\mathbf{v}}_c$. (b) how to increase the zebra score $={\nabla }_{{\mathbf{h}}_{\ell }}{S}_k(\mathbf{x})$ (the steepest-uphill arrow). (c) the raw class score (the logit) $=S_k(\mathbf{x})$.

(ii). Since TCAV counts the fraction of positive dot products, a score of $0$ means no example had the gradient pointing toward the concept — the model ignores or actively opposes it. The random baseline is $\approx 0.5$ (see the symmetry argument above), not $0$.

concept ::: direction of "more of this concept" usage ::: does the model actually rely on it to decide presence ::: whether the concept merely appears in the image

${\nabla }_{{\mathbf{h}}_{\ell }}{S}_{\text{zebra}}\cdot {\mathbf{v}}_{\text{striped}}=3(1)+(-1)(0)+2(1)=3+0+2=5.$ $5>0$, so $1\{5>0\}=1$: it votes yes — nudging activations toward "striped" would raise the zebra logit.

$\text{TCAV}=\frac{87}{100}=\mathrm{0.87.}$ The model uses "striped" for the vast majority of zebra examples.

Dot product: $\mathbf{g}\cdot {\mathbf{v}}_c=2(-1)+2(1)=-2+2=0$. Since $\mathbf{a}\cdot \mathbf{b}=\Vert \mathbf{a}\Vert \Vert \mathbf{b}\Vert \cos \theta$ and the product is $0$ with non-zero lengths, $\cos \theta =0\Rightarrow \theta =90^{\circ }$. The arrows are perpendicular: this example sits exactly on the fence. Since the vote requires strictly $>0$, it is not a yes vote (a boundary example counts as no).

We measure the gap in units of the random distribution's own spread (a z-score): $z=(\text{value}-\text{mean})/\text{std}$. $z=\frac{0.87-0.52}{0.08}=\frac{0.35}{0.08}=4.375\approx 4.4\sigma .$ Caveat on interpretation: turning $4.4\sigma$ into a tiny $p$-value assumes the random scores are roughly bell-shaped (Gaussian) — a convenient approximation, not a proven fact. But the gap is so large that even without that assumption it dwarfs the observed spread, so the conclusion is safe: the concept is genuinely used, not a fluke of high-dimensional geometry. (Ex 3.2 shows the assumption-free way to get a $p$-value by direct counting.)

$p=\frac{1+6}{20+1}=\frac{7}{21}=\frac{1}{3}\approx \mathrm{0.3333.}$ $0.333\nless 0.05$, so this concept fails the significance test — $0.60$ is well within random reach. Reject the claim that the model uses it. (This count-based $p$ needs no Gaussian assumption — it just asks how often random directions matched or beat you.)

The gradient rarely points toward stripes ($0.30<$ baseline $0.5$) but almost always points toward grass ($0.90$). The model is classifying zebras largely by their background — a spurious shortcut. This is a dataset bug: every zebra image shared the grassland context, so "grass" was a perfectly predictive but semantically wrong cue. This is exactly the kind of shortcut surfaced in Adversarial Examples and repairable via Model Editing.

(a) Sets: P = image patches containing the watermark (crop many corners bearing it); N = random patches without the watermark, matched in count. (b) Layer $\ell$: a mid-to-late convolutional layer, where features are abstract enough to encode a small textural mark but before the final logits so a gradient still flows meaningfully. Fix this $\ell$ for the whole experiment. (c) Class: $k=$ "pneumonia". (d) A high, significant $\text{TCAV(watermark, pneumonia)}$ proves the gradient of the pneumonia logit consistently points toward the watermark direction — i.e. the model uses the watermark to decide, a catastrophic shortcut (some hospitals watermarked mostly sick patients). This connects to Probing Classifiers: probing would only tell you the watermark is encoded; TCAV tells you it is used.

ACE is working correctly — it faithfully found a direction the model uses. The finding itself is the alarm: the model classifies "cow" partly via grass background. ACE's strength is discovering concepts you never thought to test (you would likely have hand-tested "hooves", "horns", "cow-body", never "grass"). The lesson: automated discovery + TCAV together surface hidden spurious concepts that manual concept sets would miss.

A vote is $1\{\mathbf{g}\cdot {\mathbf{v}}_c>0\}$. Scaling: $(\alpha \mathbf{g})\cdot {\mathbf{v}}_c=\alpha (\mathbf{g}\cdot {\mathbf{v}}_c)$. For $\alpha >0$, multiplying by a positive number preserves the sign, so $1\{\alpha (\mathbf{g}\cdot {\mathbf{v}}_c)>0\}=1\{\mathbf{g}\cdot {\mathbf{v}}_c>0\}$. Vote unchanged. ∎ Why it is a feature: logit gradients have arbitrary magnitude depending on network scale, layer, and normalisation. If TCAV depended on magnitude it would be uncomparable across layers/models. By keeping only the sign, TCAV measures the geometric alignment of the concept with the "which way is uphill" direction — a scale-free, interpretable quantity.

(a) $\mathbf{g}\cdot {\mathbf{v}}_c=0$, so $1\{0>0\}=0$: counts as no. Interpretation: at a saturated point the class score cannot be increased at all, so no direction — concept or otherwise — helps. Beware: many saturated examples can artificially depress TCAV; check gradient norms. (b) $\mathbf{g}\cdot (-\mathbf{g})=-\Vert \mathbf{g}{\Vert }^2<0$: no. The concept direction points exactly downhill for the class — moving toward the concept lowers the class score. This is a strong "concept opposes class" signal. (c) dot product $=0$, vote is no (boundary counts as no, matching the strict $>0$). The concept is irrelevant to first order for this example.

Both can be correct because they measure different questions (see Probing Classifiers):

• Probing at 92% shows texture is linearly decodable — the information is present and encoded in the activations.
• TCAV $\approx 0.48$ (essentially the random baseline of $0.5$) shows that, at decision time, the class logit's gradient is not aligned with the texture direction — the model does not use texture to make this decision. Reconciliation: a network can represent a feature richly yet route its decision through other features entirely. The texture information lives in the activations (so a probe reads it easily), but the class logit does not rise when you push activations toward the texture direction (so TCAV stays at chance). In one sentence: encoded ≠ used. This is the single most important distinction in concept-based interpretability, and why causal directional-derivative testing (TCAV) is not replaceable by probing accuracy alone. Compare with Feature Attribution Methods and Layer-wise Relevance Propagation, which answer yet another question ("which inputs mattered") entirely.

The whole page reduces to ::: TCAV = fraction of class examples whose logit-gradient points to the same side as the concept arrow; only the SIGN of the dot product matters; and this must beat a random-CAV baseline (≈0.5 by symmetry) to count.