Probing classifiers: Visual walkthrough — AI-ML deep dive

Prerequisite ideas we lean on live in Probing classifiers (the parent), and it will help to have met Attention Mechanisms (where the representations $h$ come from) and Fine-tuning vs. Feature Extraction (why we freeze the model).

Step 1 — A representation is just a point in a room

WHAT. A neural network, when it reads a word like "cat", does not store a word — it stores a list of numbers. Say the list has $d$ numbers. We write it $h = (h_{1}, h_{2}, \dots, h_{d})$ . That bold-h is called an activation or a representation.

WHY. We cannot argue about "what the model knows" until we agree on where that knowledge lives. It lives as a location. If $d = 2$ , the location is a dot on a flat sheet of paper: $h_{1}$ is how far right, $h_{2}$ is how far up. If $d = 3$ it's a dot in a room. Real networks use $d$ in the hundreds — same idea, more directions than we can draw, so we always draw $d = 2$ and trust the picture generalizes.

PICTURE. Below, three different sentences each become one black dot. The two axes are the two numbers in the representation. A "representation" is nothing more mysterious than an address.

Step 2 — A property colours the points

WHAT. Now attach a label to each dot. For the property $τ =$ subject–verb agreement, each dot's label is either agree or disagree. We colour every agree dot with our accent colour and leave every disagree dot black.

WHY. The whole question of probing — "did the model encode the property $τ$ ?" — becomes a question about colour geometry: are the red dots and black dots in separate regions, or are they hopelessly mixed? If they sit in separate regions, the property is "in there" as a shape you could point at.

PICTURE. Same room as Step 1, now the dots are coloured by their property label. Notice the reds have drifted to one side. That drift is the model encoding the property.

Step 3 — The linear probe is a straight cut

WHAT. A linear probe tries to separate the two colours with a single straight cut — a line in 2-D, a flat sheet (a hyperplane) in higher $d$ . The cut is described by an arrow $w$ (which way is "more red") and an offset $b$ (how far from the origin the cut sits):

direction of the cut w^{⊤} a point h + shift b = 0.

$w^{⊤} h$ ::: the dot product, which measures how far along the $w$ direction the point $h$ sits.
$= 0$ ::: the exact set of points sitting on the cut.
$> 0$ on one side, $< 0$ on the other side.

WHY a straight cut and not a wiggly one? Because we are testing whether the information is easy to read out. A wiggly boundary (an MLP) could carve out any clump of reds no matter how scrambled — so it would tell us nothing about whether the model organised the property. Insisting on a straight cut is what makes a high score meaningful: only a genuinely organised representation can be split by a plane.

WHY the dot product? We need one number per point that says "which side, and how far?" The dot product $w^{⊤} h = ∥ w ∥ ∥ h ∥ cos ϑ$ is precisely the length of $h$ 's shadow along $w$ . It answers the question "project this point onto the red-ward direction — where does it land?" No other simple operation gives a clean side-and-distance in one scalar.

PICTURE. The accent line is the cut. The accent arrow $w$ points from the cut toward the red side. Every point's shadow onto $w$ is its score $w^{⊤} h + b$ .

Step 4 — Turning a score into a probability, then a class

WHAT. The score $s = w^{⊤} h + b$ runs from $- \infty$ to $+ \infty$ . We squash it into a probability with the sigmoid $σ$ :

\overset{y}{^} = σ (s) = \frac{1}{1 + e ^{- s}}, ⎩ ⎨ ⎧ s \to + \infty s = 0 s \to - \infty \Rightarrow \overset{y}{^} \to 1 (sure red) \Rightarrow \overset{y}{^} = 0.5 (on the fence) \Rightarrow \overset{y}{^} \to 0 (sure black) .

From probability to a hard class. The accuracy in the top definition needs a discrete prediction, not a probability. We threshold at $0.5$ — which, since $σ (s) > 0.5 ⟺ s > 0$ , is the same as reading off the sign of the score:

\overset{y}{^}_{class} = {+ 1 (red) - 1 (black) if \overset{y}{^} = σ (s) \geq 0.5 ⟺ s \geq 0 if \overset{y}{^} < 0.5 ⟺ s < 0.

A point is correctly classified exactly when $\overset{y}{^}_{class} = y_{i}$ , i.e. when the true label and the side of the cut agree. The accuracy $A_{τ}^{(l)}$ is just the fraction of held-out points for which this holds.

WHY the sigmoid and not just "sign of $s$ "? Two reasons. (1) We want a smooth readout so training has a slope to follow. The hard function $sign (s)$ is not differentiable at $s = 0$ — it jumps from $- 1$ straight to $+ 1$ with no slope there and zero slope everywhere else, so gradient descent has nothing to follow. The sigmoid is smooth everywhere. (2) The value at $s = 0$ being exactly $0.5$ matches the cut $s = 0$ being the "no idea" boundary from Step 3. So $σ$ turns the geometry (which side, how far) into confidence — and thresholding turns that confidence back into a class for scoring.

PICTURE. The S-shaped curve: horizontal axis is the raw score $s$ , vertical axis is the probability. The accent point marks $s = 0 \Rightarrow \overset{y}{^} = 0.5$ , sitting right on the cut; the dashed threshold at $0.5$ shows where the class prediction flips.

Formula The probe's training objective (why smoothness matters concretely)

We fit $w, b$ by minimising the regularised cross-entropy (logistic) loss over the $N$ probe points:

L (w, b) = - \frac{1}{N} i = 1 \sum N [t_{i} lo g σ (s_{i}) + (1 - t_{i}) lo g (1 - σ (s_{i}))] + λ ∥ w ∥^{2}, s_{i} = w^{⊤} h_{i} + b .

$t_{i} \in {0, 1}$ ::: the $0/1$ version of the label ( $1$ = red, $0$ = black) that cross-entropy expects.
$- [t_{i} lo g σ + (1 - t_{i}) lo g (1 - σ)]$ ::: the penalty for being confidently wrong — it blows up as $σ (s_{i}) \to 0$ for a red point. Because $σ$ is smooth, this whole sum has a gradient $\partial L / \partial w$ that gradient descent can follow — this is the concrete payoff of Step 4's smoothness.
$λ ∥ w ∥^{2}$ ::: the L2 regularizer. It adds cost proportional to the squared length of $w$ , so its gradient $2 λ w$ always pulls $w$ toward $0$ . Bigger $λ$ = stronger pull = simpler (shorter) cut. Set $λ$ far too high and this pull wins outright, collapsing $w \to 0$ (the degeneracy of Step 7).

Step 5 — The margin: how far the nearest point sits from the cut

WHAT. Many straight cuts might separate the colours. Which is "best"? The one that leaves the biggest empty gap. The margin $γ$ is the distance from the cut to the closest point:

γ = i min \frac{y _{i} ( w ^{⊤} h _{i} + b )}{∥ w ∥} .

$w^{⊤} h_{i} + b$ ::: point $i$ 's signed score (Step 3).
$y_{i} \in {+ 1, - 1}$ ::: the numeric label from Step 2 ( $+ 1$ red, $- 1$ black); multiplying by it makes a correctly placed point give a positive product (right sign × right side = $+$ ).
$\div ∥ w ∥$ ::: divides out the arrow's length so $γ$ is a true distance in the room, not just a raw score (doubling $w$ must not "double" the margin — it's the same cut).
$min_{i}$ ::: we care about the worst (closest) point — that's the weakest link.

WHY care about the gap? A wide gap means the reds and blacks are in firmly separate territories — the property is robustly, unmistakably encoded. A razor-thin gap means the probe barely works: a whisker of noise flips points across. So margin size measures how confidently the layer encodes $τ$ , beyond just "did the cut succeed."

WHY margin controls accuracy on unseen points. Held-out accuracy $A_{τ}^{(l)}$ is about points the probe never trained on. Picture wrapping a tube of radius $γ$ around the cut: no training point lies inside it. A fresh test point that is a genuine sample of the same colours must land near its own colour's cloud; to be misclassified it would have to jump all the way across that empty tube. The wider the tube (bigger $γ$ ), the more a test point can wander and still stay on the correct side — so larger margin ⇒ fewer test-time flips ⇒ higher $A_{τ}^{(l)}$ . This is the intuition behind the classical bound "test-error $≲$ (something) $/ γ^{2}$ ": generalization error shrinks as the margin grows.

PICTURE. Two parallel dashed lines hug the cut; the accent double-arrow between them is $2 γ$ . The circled points touching the dashed lines are the closest points that set the margin. The shaded empty tube is the room a test point may wander in without being misclassified.

Step 6 — Edge case A: the colours overlap (no straight cut exists)

WHAT. Sometimes no straight line separates the colours — a red dot sits deep in black territory or vice-versa. Then the best line still makes mistakes.

WHY show this? The parent said "low accuracy means the info is absent or requires complex nonlinear extraction." This is the picture of that sentence. When points overlap, a straight cut cannot score $100%$ no matter how you place it. Two very different situations produce this:

Absent: the layer genuinely didn't encode $τ$ — reds and blacks are fully mixed, like salt and pepper.
Nonlinearly encoded: the reds sit in a ring around the blacks. A straight cut fails, yet a curved boundary (an MLP probe, Disentangled Representations) would succeed. The info is present but entangled.

PICTURE. Left half: salt-and-pepper mix — even the best straight cut misclassifies ~half. Right half: a red ring around a black core — no line works, but the accent dashed curve does. This is exactly why Worked Example 2 saw MLP $≫$ linear.

Step 7 — Edge case B: a degenerate cut (all one colour, or $w = 0$ )

WHAT. Two degeneracies to name so you're never surprised:

One colour only. If every point in $D_{probe}$ is red, a probe that always says "red" scores $100%$ — but has learned nothing. This is class imbalance, and accuracy alone lies.
The zero cut, $w = 0$ . Then $w^{⊤} h + b = b$ for every point — the score ignores $h$ entirely. The "cut" collapses; there's no direction, so the probe just outputs the majority class.

WHY. Both are traps that inflate accuracy without any real encoding. The fix for (1) is a majority-class baseline: only accuracy above the baseline counts. Case (2) is what the L2 regularizer $λ ∥ w ∥^{2}$ from Step 4 can accidentally cause: recall its gradient $2 λ w$ pulls $w$ toward $0$ . Here $λ \geq 0$ is a hyperparameter (a knob we choose before training). A little $λ$ is healthy — it stops the probe memorising noise. But too much $λ$ crushes $w$ toward $0$ , collapsing the cut and measuring nothing. So $λ$ must be tuned, not maxed out.

PICTURE. Left: all-red cloud with a "trivial 100%" flag. Right: $w = 0$ , the accent shows a collapsed cut that every point sits on the same side of — no separation possible.

Step 8 — Putting numbers on it: reproducing the layer trend

WHAT. The parent reported BERT POS-probe accuracy climbing then dipping across three depths: $A_{POS}^{(0)} = 82%$ at layer $0$ (raw word embeddings), $A_{POS}^{(6)} = 97%$ at the middle layer $6$ , and $A_{POS}^{(12)} = 95%$ at the final layer $12$ . Read off the picture, this is the margin widening into the middle layers, then shrinking slightly at the output.

WHY the shape? Early layers ( $l = 0$ ) only know word identity — reds and blacks partly overlap (margin small, so $A_{POS}^{(0)} = 82%$ ). Middle layers ( $l = 6$ ) add context that spreads the colours apart (margin widest, $A_{POS}^{(6)} = 97%$ , the peak). The output layer ( $l = 12$ ) re-tangles things to serve the training objective, so the gap closes a touch ( $A_{POS}^{(12)} = 95%$ ). The trend is non-monotone: it rises from $82$ to $97$ , then dips to $95$ — the peak is in the middle, not at the end.

PICTURE. Accuracy $A_{POS}^{(l)}$ vs. layer $l$ , with the accent peak at the middle layer $l = 6$ . Below each dot, a mini-cloud shows the margin: narrow → wide → narrow, mirroring the accuracy numbers.

The one-picture summary

Everything above collapses into a single flow: input → representation (a point) → colour it by the property $τ$ → find the widest straight cut → threshold the sigmoid into a class → the margin measures how confidently the property is encoded and how high held-out $A_{τ}^{(l)}$ climbs → if no cut works, it's either absent or curved (entangled).

Recall Feynman retelling — say it back in plain words

A network is a stack of stages called layers, numbered $l$ ; each stage turns the sentence it reads into a dot in a big invisible room — that dot is just the list of numbers the stage scribbled down. To ask "did layer $l$ learn about, say, subject–verb agreement (my property $τ$ )?", I paint every agree dot red (label $+ 1$ ) and every disagree dot black (label $- 1$ ). Then I try to slide a single straight ruler between the colours. If one clean slice keeps all reds on one side and all blacks on the other, the network organised the property — it's linearly in there. The dot product is just each dot's shadow onto the "red-ward" arrow, telling me which side and how far. I squash that distance through the smooth S-curve to get a confidence, then I threshold at $0.5$ (same as reading the sign) to get a hard red/black guess — and the fraction of held-out dots I guess right is the accuracy $A_{τ}^{(l)}$ . I actually find the ruler by minimising a cross-entropy loss (smooth, so gradient descent has a slope to chase) with a little $λ ∥ w ∥^{2}$ penalty that gently shrinks the ruler's arrow toward zero. The margin is the empty tube around the ruler — a wide tube means fresh test dots can wander a lot and still land on the right side, so accuracy stays high; a paper-thin tube means a whisker of noise flips them. If the colours are salt-and-pepper mixed, either the network never learned it, or — the sneaky case — the reds form a ring around the blacks, so no straight ruler works but a curved one does; then the info is there but tangled up. If all my dots are one colour, a lazy "always red" ruler scores 100% while knowing nothing — so I only trust a probe that beats the majority-guess baseline. Layer by layer the margin (and so $A_{τ}^{(l)}$ ) rises into the middle then dips at the end.

Recall Checklist of every case we covered

Separable, wide margin ::: explicit linear encoding — high linear accuracy $A_{τ}^{(l)}$ Separable, thin margin ::: barely-there encoding, test points flip, $A_{τ}^{(l)}$ near baseline Overlapping (salt & pepper) ::: information absent Overlapping but curved-separable (ring) ::: present but entangled — MLP beats linear All one colour ::: trivial 100%, must check majority baseline $w = 0$ (over-regularised, big $λ$ ) ::: cut collapses, measures nothing Many labels ::: one-vs-rest hyperplanes or softmax over $K$ classes