Visual walkthrough — Concept-based explanations

---

Step 1 — What is "activation space"?

WHAT. Pick one hidden layer inside the network, call it layer $\ell$ (the Greek letter ell, just a name for "which layer"). When you push an image through the net and pause at that layer, the layer emits a list of numbers. If the layer has $d$ neurons, that list is $d$ numbers long. We write it as one arrow (a vector) and call it

${\mathbf{h}}_{\ell }=(\underset{\text{neuron 1}}{\underbrace{h_1}},\underset{\text{neuron 2}}{\underbrace{h_2}},\dots ,\underset{\text{neuron }d}{\underbrace{h_d}}).$

Every image becomes one point in a $d$-dimensional space. We cannot draw 2048 dimensions, so every figure on this page draws just 2 of those neurons as the two axes — the picture that fits on paper. The logic is identical in high dimensions.

When we have several images and need to talk about "the activation of image number $i$," we hang a little label on top: ${\mathbf{h}}_{\ell }^{(i)}$ means the activation vector produced by the $i$-th image at layer $\ell$. Same object as ${\mathbf{h}}_{\ell }$, just tagged with which image it came from.

WHY a vector and not a picture of pixels? Because the model's own understanding lives here, not in the pixels. Two images that look different (a tiger, a striped shirt) can land near each other in activation space if the model "thinks about them" the same way.

PICTURE. Each dot is one image's activation.

---

Step 2 — Two clouds: concept vs. random

WHAT. Gather two sets of images:

• $P$ (positive) — images that contain the concept, e.g. striped things.
• $N$ (negative/random) — ordinary images with no particular concept.

Feed all of them through, stop at layer $\ell$, and drop every activation ${\mathbf{h}}_{\ell }^{(i)}$ (from Step 1) as a dot. We label each dot:

$y_i=\{\begin{array}{ll}+1& \text{if image }i\in P(\text{striped})\\ -1& \text{if image }i\in N(\text{random}).\end{array}$

Here $y_i$ (read "y-sub-i") is just a tag on dot number $i$ saying which cloud it belongs to.

WHY. If the model has an internal notion of "stripes," the $P$ dots should huddle together, separate from the $N$ dots. We are about to measure that separation.

PICTURE. Orange = striped ($+1$), gray = random ($-1$).

---

Step 3 — Slice the two clouds with a straight cut

WHAT. Draw the flat surface (a hyperplane — in 2D just a straight line) that best separates orange from gray. A line is described by the direction $\mathbf{w}$ pointing across it (perpendicular to it). We find the best $\mathbf{w}$ by minimizing a training loss over the labelled dots ${\mathbf{h}}_{\ell }^{(i)}$ from Step 2:

${\mathbf{v}}_c=\arg {\min }_{\mathbf{w}}\underset{\text{penalize wrong side}}{\underbrace{\sum _{i\in P\cup N}\mathcal{L}(\underset{\text{score of dot }i}{\underbrace{{\mathbf{w}}^{\top }{\mathbf{h}}_{\ell }^{(i)}}},y_i)}}+\underset{\text{keep }\mathbf{w}\text{ small}}{\underbrace{\lambda \Vert \mathbf{w}{\Vert }^2}}.$

Term by term:

• ${\mathbf{w}}^{\top }{\mathbf{h}}_{\ell }^{(i)}$ — the dot product of the direction $\mathbf{w}$ with dot $i$ (the $i$-th activation from Step 1). It measures how far along $\mathbf{w}$ the dot sits. (Why a dot product? Because it collapses a whole vector down to one number: "which side of the line, and how far.")
• $\mathcal{L}(\cdot ,y_i)$ — the loss. Concretely, the standard choice (as in the original TCAV) is logistic (cross-entropy) loss: $\mathcal{L}(z,y)=\log (1+e^{-yz})$, where $z={\mathbf{w}}^{\top }{\mathbf{h}}_{\ell }^{(i)}$ is the score and $y=\pm 1$ the tag. Notice what it does: when $z$ has the same sign as $y$ (correct side), $e^{-yz}$ is small and the loss is near $0$; when the signs disagree (wrong side), $e^{-yz}$ is large and the loss grows. Any loss with this "small when right, large when wrong, smooth" property works (hinge loss, used by a linear SVM, is another common choice).
• $\arg {\min }_{\mathbf{w}}$ — "the $\mathbf{w}$ that makes this smallest." (Not the min value; the winning direction.)
• $\lambda \Vert \mathbf{w}{\Vert }^2$ — a regularization penalty. $\lambda$ (lambda) is a small positive number you choose (typical scale $10^{-3}$ to $1$). Its job: keep the entries of $\mathbf{w}$ from blowing up. Larger $\lambda$ → shorter, smoother $\mathbf{w}$ that ignores noisy neurons but may under-fit; $\lambda =0$ → no penalty, easy to over-fit. Since we only ever use the direction of $\mathbf{w}$, $\lambda$ mainly controls how much noise the boundary is allowed to chase.

The winning direction, once we shrink it to length 1, is the Concept Activation Vector ${\mathbf{v}}_c$.

WHY a straight cut and not a curvy one? Because we want a single, clean direction to call "the striped direction." A curve has no one direction; a line does. That direction is the whole payoff.

PICTURE. The line separates the clouds; the red arrow ${\mathbf{v}}_c$ points from gray into orange — "more striped this way."

---

Step 4 — What direction increases the class score?

WHAT. Separately, the network has an output number for "zebra," the logit $S_k(\mathbf{x})$ (score for class $k$, before the final softmax). Ask: if I could nudge the activation ${\mathbf{h}}_{\ell }$ a tiny bit, which way makes $S_k$ grow fastest? That fastest-uphill direction is the gradient:

${\nabla }_{{\mathbf{h}}_{\ell }}{S}_k(\mathbf{x})=\left(\frac{\partial S_k}{\partial h_1},\dots ,\frac{\partial S_k}{\partial h_d}\right).$

Term by term:

• ${\nabla }_{{\mathbf{h}}_{\ell }}$ (read "nabla", the upside-down triangle) — "collect the slope of $S_k$ with respect to every neuron."
• $\frac{\partial S_k}{\partial h_j}$ — a partial derivative: "if only neuron $j$ wiggles up a hair, how much does the zebra score move?" Positive = pushing that neuron up helps zebra.

WHY a gradient? Because we need a direction of steepest increase of the class score, and the gradient is exactly that — it is the arrow that answers "which way is uphill for zebra?" We get it for free by backpropagation, the same machinery that trains the net.

PICTURE. Blue arrow = gradient of the zebra logit at one zebra's activation dot; contour lines show the score climbing.

---

Step 5 — The key move: does concept-direction agree with uphill?

WHAT. Now put the two arrows in the same space:

• ${\mathbf{v}}_c$ — the "more striped" direction (Step 3, red).
• ${\nabla }_{{\mathbf{h}}_{\ell }}{S}_k$ — the "more zebra" direction (Step 4, blue).

Compare them with a dot product:

$s(\mathbf{x})=\underset{\text{uphill for zebra}}{\underbrace{{\nabla }_{{\mathbf{h}}_{\ell }}{S}_k(\mathbf{x})}}\cdot \underset{\text{toward stripes}}{\underbrace{{\mathbf{v}}_c}}.$

The dot product of two arrows equals (length)(length)(cosine of the angle between them). So its sign is the sign of that cosine:

• angle $<90^{\circ }$ → arrows agree → $s>0$ → becoming more striped also makes it more zebra.
• angle $>90^{\circ }$ → arrows oppose → $s<0$ → stripes push zebra score down.
• angle $=90^{\circ }$ → $s=0$ → stripes are irrelevant here (the degenerate, on-the-fence case).

WHY the dot product and not, say, distance? Because we don't care how big either arrow is — we care whether pushing toward the concept helps or hurts the class. Only the angle answers that, and the dot product's sign reads the angle directly.

PICTURE. Three panels: agree (green, sharp angle), oppose (red, obtuse angle), perpendicular (gray, the zero case).

---

Step 6 — From one example to a score: vote and average

WHAT. One zebra could be a fluke, so test many. Let $X_k$ be the set of test images that belong to class $k$ (e.g. a batch of held-out zebra photos — importantly not the images used to fit ${\mathbf{v}}_c$ in Step 3). For each image $\mathbf{x}\in X_k$, mark a vote:

$1\{s(\mathbf{x})>0\}=\{\begin{array}{ll}1& \text{if arrows agree (green)}\\ 0& \text{otherwise.}\end{array}$

$1\{\cdot \}$ is the indicator: it turns a yes/no question into a $1$/$0$. Then average the votes:

${\text{TCAV}}_{\ell ,c,k}=\underset{\text{average over class-}k\text{ images}}{\underbrace{{\mathbb{E}}_{\mathbf{x}\in X_k}}}\left[\underset{\text{1 if concept helps this image, else 0}}{\underbrace{1\{{\nabla }_{{\mathbf{h}}_{\ell }}{S}_k(\mathbf{x})\cdot {\mathbf{v}}_c>0\}}}\right].$

The result is just the fraction of green votes over $X_k$ — a number between $0$ and $1$.

WHY average and not sum? So the score is comparable across concepts and classes regardless of how many test images $X_k$ held — it's always a fraction.

PICTURE. A row of zebra dots, each with a tiny green/red mark; the fraction green = TCAV.

---

Step 7 — The degenerate case: why $0.5$ is "nothing", not "half"

WHAT. Take a random direction instead of a real concept — a ${\mathbf{v}}_c$ pointing nowhere meaningful. In a high-dimensional space, a random arrow lands on either side of the gradient with roughly equal chance, so its vote is basically a coin flip and TCAV $\approx 0.5$.

WHY this matters. A raw score of $0.65$ looks "high" but could be pure noise. So we run many random CAVs, collect their scores, and ask: is the real score an outlier? The p-value

$p=\frac{1+{\sum }_{i=1}^{n}1\{{\text{TCAV}}_{\text{random},i}\ge {\text{TCAV}}_c\}}{n+1}$

counts how many random directions matched or beat the real one. Term by term: the numerator's "$1+$" and denominator's "$n+1$" are a safety padding so $p$ is never exactly $0$; the sum tallies random scores that reach the real score. Small $p$ (say $<0.05$) = the concept genuinely stands out.

PICTURE. Histogram of random-CAV scores clustered at $0.5$; the real striped score at $0.87$ sits far in the right tail.

---

The one-picture summary

Everything above is one pipeline: two clouds → a cutting line → its normal (${\mathbf{v}}_c$) → compare its angle to the class gradient → vote → average → sanity-check against random.

Recall Feynman retelling (plain words)

Deep inside the network, every picture becomes a dot in a big space of numbers. I collect a pile of striped pictures and a pile of random ones, and I slice between the two piles with a straight cut. The arrow that points straight across that cut, from random into striped, I name the "striped direction."

Separately, for a zebra picture, I ask the network: "which way should I nudge your inner numbers to make you more sure it's a zebra?" That's another arrow — the uphill-for-zebra direction.

Then the whole trick is just comparing two arrows. If the striped direction and the more-zebra direction point roughly the same way (angle under 90°), then stripes help the zebra verdict — I score a green vote. I do this for a hundred fresh zebras (not the ones I built the striped direction from) and count the greens. Say 87 out of 100 → score 0.87.

But maybe any random arrow scores high by luck. So I repeat with a bunch of nonsense directions. They average around 0.5 — a coin flip. My real 0.87 is way out in the tail, so I trust it: the network really leans on stripes to call a zebra a zebra.

Recall Check yourself

Why do we take the sign of the dot product rather than its size? ::: Because we only care whether the concept helps or hurts the class score, which is the angle between the arrows; size (lengths of the arrows) is irrelevant to that yes/no. What does TCAV $=0.5$ mean? ::: The concept behaves like a random direction — no consistent effect on the class. It is the "nothing happening" baseline, not "half important." Why must $X_k$ and the CAV-training set be disjoint? ::: Otherwise you fit and evaluate the concept direction on the same points, so a lucky over-fit boundary inflates the score; a held-out split keeps the test honest. What breaks if you skip the random-CAV test? ::: A noisy score like 0.65 could be pure high-dimensional chance; without the random baseline and p-value you cannot tell signal from noise.

---

Connections

• Feature Attribution Methods — attribution finds uphill directions in input space; TCAV finds them in concept space.
• Probing Classifiers — Step 3's linear cut is exactly a probe; TCAV then adds Steps 4–6 to test usage, not just encoding.
• Layer-wise Relevance Propagation — an alternative way to score influence, traced to inputs rather than concepts.
• Model Editing — once ${\mathbf{v}}_c$ is known, you can ablate that direction to remove a concept's influence.
• Adversarial Examples — the same gradient of $S_k$ that TCAV uses is what adversarial attacks exploit.