Probing classifiers: Worked examples — AI-ML deep dive

This is the drill page for Probing classifiers. The parent note told you what a probe is: a tiny classifier (like logistic regression) trained on frozen activations $h$ to predict some property $τ$ . Here we run that machine through every kind of situation you will ever face — clean cases, edge cases, and the traps that fool people.

Before we start, let me re-earn every symbol so line one makes sense on its own:

Definition The symbols we will reuse

$h$ (bold h) = a representation vector: the list of numbers a hidden layer spits out for one input. Picture an arrow in space; each coordinate is one neuron's activation.
$w$ (bold w) = the probe's weight vector: another arrow. The probe scores an input by asking "how much does $h$ point the same way as $w$ ?"
$w^{⊤} h$ = the dot product: multiply matching coordinates, add them up. It's a single number measuring alignment. Big positive = same direction; negative = opposite; zero = perpendicular.
$b$ (plain b) = the bias: a single number the probe adds after the dot product to shift the boundary off the origin. The full score is $s = w^{⊤} h + b$ . Picture sliding the dividing line left or right without rotating it.
$y$ (plain y, or $y_{i}$ for the $i$ -th example) = the true label we want the probe to output — the correct answer. In a two-class task we write it as $+ 1$ or $- 1$ so that "label times score" is positive exactly when the guess is right.
$A_{τ}^{(l)}$ = probe accuracy for task $τ$ at layer $l$ : the fraction of held-out examples the probe labels correctly, from $0$ (never right) to $1$ (always right).
$γ$ (Greek "gamma") = the margin: the distance from the boundary to the nearest data point. Picture the widest empty gutter you can slide the boundary down before it touches a point. We compute it as $γ = min_{i} \frac{y _{i} ( w ^{⊤} h _{i} + b )}{∥ w ∥}$ . $γ > 0$ means every point is on its correct side with room to spare; $γ = 0$ means a point sits on the boundary; $γ < 0$ means at least one point is misclassified. ( $∥ w ∥$ is the length of the arrow $w$ ; dividing by it turns the raw score into a true distance.)
$ϕ$ (Greek "phi") = a feature map: a fixed function we apply to $h$ before the probe, e.g. $ϕ (h) = h^{2}$ . Picture bending the number line so that points that were tangled become separable. It lets a linear probe act on the transformed values.

Why the dot product and not something fancier? A linear probe is deliberately weak. If a weak tool can read $τ$ off $h$ , the information must sit right at the surface — that is exactly the question probing asks.

The scenario matrix

Every probing situation is one of these cells. The worked examples below each carry a tag like [Cell A1] so you can see we left no gap. Cells are numbered $0, 1, 2$ inside each letter group in increasing difficulty, so the drill reads top-to-bottom.

Cell	Situation	What makes it tricky
A1	Clean linear separation, positive margin ( $γ > 0$ )	The easy baseline — probe wins big
A2	Separable but zero margin ( $γ = 0$ )	Points sit on the boundary — knife-edge case
A3	Info present but only nonlinearly	Linear probe fails, feature map succeeds
B1	Zero / degenerate input: property absent	Accuracy collapses to the majority-class baseline
B2	Degenerate: representation is constant ( $h$ same for all)	Probe can't separate anything — chance level
C1	Limiting behaviour across layers (early → deep)	Accuracy curve rises then may fall
C2	Limiting: selectivity (control task) near zero	High accuracy but the probe memorised — a false positive
D1	Sign / direction flip in $w$	Same boundary, mirrored labels — must handle both signs
E1	Real-world word problem	Full study from raw numbers
F1	Exam-style twist	Probe beats the model itself — what does it mean?

We now hit each cell.

Worked examples

Example 1 — the clean baseline [Cell A1, D1]

Forecast: Guess — will the boundary sit near $h = 0$ ? Positive or negative margin?

Figure s01 plots the four points on a single horizontal number line (the 1-D representation axis $h$ ). The two red dots at $h = - 2$ and $h = - 1$ are the $y = - 1$ (singular) class; the two blue dots at $h = + 1$ and $h = + 3$ are the $y = + 1$ (plural) class. The vertical yellow line at $h = 0$ is the learned boundary $w h + b = 0$ . The green arrow marks the gap of size $1$ from the boundary to the nearest point $h = 1$ — that gap is the margin $γ$ .

Steps.

Pick the boundary. The two negatives sit at $h \leq - 1$ , the two positives at $h \geq + 1$ . A boundary at $h = 0$ splits them. Set $w = 1, b = 0$ so the score is $s = w h + b = h$ . Why this step? The decision rule is "predict $+ 1$ when $s > 0$ ". We just need any hyperplane (here a single point on the line) that puts each label on its correct side. Since $b = 0$ the boundary sits exactly at the origin.
Check each point's signed score $y_{i} (w h_{i} + b)$ . This product is positive exactly when the prediction is correct (right sign matches label). $y (w h + b) : (- 1) (- 2) = 2, (- 1) (- 1) = 1, (+ 1) (1) = 1, (+ 1) (3) = 3.$ Why this step? All four are $> 0$ ⇒ every point is on the correct side ⇒ linearly separable.
Compute the margin $γ = min_{i} \frac{y _{i} ( w h _{i} + b )}{∥ w ∥}$ . Here $∥ w ∥ = ∣1∣ = 1$ , and the smallest numerator is $1$ . $γ = \frac{1}{1} = 1.$ Why this step? The margin is the distance from the boundary to the nearest point. $γ > 0$ confirms a robust, easy linear encoding — the hallmark of a good probe result.
Sign flip (Cell D1). If we instead learned $w = - 1, b = 0$ , the score flips sign and every point is now misclassified. Flipping the labels' meaning (call $+ 1$ "singular") restores correctness. Moral: the probe direction $w$ and the label convention must agree; the geometry (the boundary at $h = 0$ ) is identical. Why this step? It shows both signs are legal descriptions of the same separating plane — you must fix a convention.

Verify: With $w = 1, b = 0$ all 4 points classify correctly ⇒ accuracy $A = 4/4 = 1.0$ . Margin $= 1 > 0$ . ✔

Example 2 — separable but zero margin [Cell A2]

Forecast: If a point sits on the line, is its signed score positive, negative, or zero?

Steps.

Place the boundary on a point. Choose $w = 1, b = 0$ , so the boundary is at $h = 0$ — right on top of the $y = + 1$ point at $h = 0$ . Why this step? We are deliberately constructing the degenerate case the margin definition must handle: a data point lying on the decision surface.
Signed score of the on-boundary point. $y (w h + b) = (+ 1) (1 \cdot 0 + 0) = 0$ . Why this step? The point contributes a numerator of $0$ , so the minimum over all points is $0$ : $γ = 0$ .
What $γ = 0$ means. The data is still separable in the weak sense (no point is on the wrong side), but there is no cushion. Any tiny wiggle in $h$ flips this point's prediction — the probe is fragile. A logistic-loss probe would keep growing $∥ w ∥$ to push this point off the line, which is exactly why we add L2 regularization. Why this step? It closes the gap in the margin story: $γ > 0$ (robust, Ex 1), $γ = 0$ (knife-edge, here), $γ < 0$ (misclassified, impossible for separable data).

Verify: $(+ 1) (1 \cdot 0 + 0) = 0$ , and the minimum signed score over the four points is $0$ ⇒ $γ = 0$ , not positive. ✔

Example 3 — present but nonlinear [Cell A3]

Forecast: A single threshold on a line — can it fence off the middle from the outsides?

Figure s02 shows the same 1-D axis. Blue dots at $h = - 3$ and $h = + 3$ are the agent class (sitting on the two outer flanks); red dots stacked at $h = 0$ are the patient class (trapped in the middle). Two dashed yellow lines show that any single linear cut leaves agents on both sides — no cut isolates the red middle. The caption reminds us we then square the axis to fix this.

Steps.

Try linear. A linear rule is "predict agent when $w h + b > 0$ ". Whatever line we draw, it cuts the axis into two rays. But agents live on both outer rays and patients in the middle — no single cut isolates the middle. Why this step? This is the XOR-shaped trap: linearly inseparable even though the classes are perfectly distinct.
Best a linear probe can do — counted exactly on the 6 samples. Any threshold leaves at least one class straddling it. Take the boundary at $h = - 1.5$ : it labels $h = - 3$ (agent) as one side and everything $\geq - 1.5$ (the three patients at $0$ and the agent at $+ 3$ ) as the other. The best labelling then gets the 3 patients + 1 agent correct on the majority side and the 1 agent at $- 3$ correct → but the $+ 3$ agent is forced to share the patient label. Counting the 6 points, the maximum correct is $4$ , giving $A_{lin} = 4/6 = 0.667$ . Why this step? With the dataset now pinned down (6 balanced samples), the $2/3$ figure is exact, not hand-wavy.
Add an explicit nonlinear feature map. Apply the feature map $ϕ (h) = h^{2}$ first, then probe linearly on $ϕ (h)$ . Now agents map to $9$ , patients to $0$ . A boundary at $ϕ = 4.5$ separates them perfectly. Why this step? A linear probe reading the hand-crafted feature $ϕ (h) = h^{2}$ nails it. In practice you don't get to choose $ϕ$ ; you hope the network's deeper layers already contain such a coordinate. Caveat on MLPs: a ReLU MLP does not reproduce $h^{2}$ exactly — ReLU is piecewise-linear, so it can only build a piecewise-linear approximation of a parabola (more hidden units = finer approximation). The point stands: nonlinear probes succeed where linear ones fail. A big MLP-vs-linear gap = "info is here but entangled" — see Disentangled Representations.

Verify: After squaring, agents $= 9 > 4.5$ , patients $= 0 < 4.5$ ⇒ all 6 correct, accuracy $= 1.0$ . Linear best $= 4/6 \approx 0.667$ . ✔

Example 4 — property absent, majority baseline [Cell B1]

Forecast: Zero? 50%? Something set by the label proportions?

Steps.

What "absent" means. Random activations carry no reliable signal about POS. The best any classifier can do is ignore $h$ and always guess the most common label. Why this step? Establishes the floor — a probe can never do worse than the majority class on average.
Compute the majority baseline. Always predict "noun": correct on 70% of samples. $A_{baseline} = max (0.70, 0.30) = 0.70.$ Why this step? $70%$ sounds impressive but is worthless here — it's pure class imbalance, not encoded information.
The lesson. Never report raw accuracy alone. Report accuracy above baseline: $0.70 - 0.70 = 0$ . Zero gain ⇒ no information about $τ$ . Why this step? Guards against the classic "high number = model understands" fallacy from the parent's Mistake 1.

Verify: $max (0.7, 0.3) = 0.7$ ; gain over baseline $= 0$ . ✔

Example 5 — constant representation, chance level [Cell B2]

Forecast: If every input looks identical, can any classifier tell them apart?

Steps.

Score is constant. $s = w^{⊤} (1, 1) + b = w_{1} + w_{2} + b$ — the same number for all inputs. Why this step? A single fixed score means every input gets the same predicted label.
Best strategy. Predict whichever label is more common. On a 50/50 split that's just $0.50$ . $A = max (0.5, 0.5) = 0.50.$ Why this step? Confirms the theoretical chance floor for a balanced binary task.

Verify: Balanced binary majority baseline $= 0.5$ . ✔ Mutual information $I (h; Y) = 0$ because $h$ is constant. ✔

Example 6 — layer-wise limiting curve [Cell C1]

Forecast: Monotone up? Or rise-then-dip?

Figure s03 is a line plot: the horizontal axis is the layer index $l \in {0, 6, 12}$ , the vertical axis is POS probe accuracy $A$ . The blue curve rises steeply from $0.82$ at layer 0 to a yellow-highlighted peak of $0.97$ at layer 6 (circled in green), then dips slightly to $0.95$ at layer 12. A dashed red line marks the multi-class uniform-chance baseline $1/17 \approx 0.059$ far below, showing all layers crush chance.

Steps.

Read the trend. From layer 0 to 6 accuracy climbs ( $+ 0.15$ ); from 6 to 12 it dips ( $- 0.02$ ). Why this step? Establishes the classic rise-then-fall signature: middle layers hold the most explicit syntax.
Find the argmax. The peak is at $l = 6$ with $A = 0.97$ . Why this step? $ar g max_{l} A^{(l)} = 6$ pinpoints where the property is most linearly accessible — the actionable output of a probing study.
Get the baseline right — this is multi-class. POS is not binary. With $17$ tags, uniform random guessing gives $1/17 \approx 0.059$ , not $0.50$ . (If you always guessed the single most common tag, say nouns at $\approx 27%$ , the majority baseline would be $\approx 0.27$ — still far below $0.97$ .) Either way, all layers are vastly above chance. Why this step? Using $0.50$ as the baseline for a 17-way task would badly understate how impressive $0.97$ is — the baseline must be $1/# classes$ (uniform) or the majority-class rate.
Explain the fall. Output layers specialise toward the pre-training objective (masked-token prediction), partially discarding task-irrelevant syntax — an attention-driven re-focus. Why this step? Interprets the limit $l \to L$ : deeper isn't always more informative.

Verify: $max (0.82, 0.97, 0.95) = 0.97$ at index 6; slope 0→6 positive, 6→12 negative; uniform chance $1/17 \approx 0.0588$ . ✔

Example 7 — control task / selectivity [Cell C2]

Forecast: Is $0.94$ trustworthy? What does an $0.85$ on random labels imply?

Steps.

Define selectivity. $Sel = A_{τ} - A_{ctrl}$ . It subtracts off how much the probe can fit nonsense — its raw memorisation power. Why this step? A probe that scores high on shuffled labels is just memorising word identities, not reading real structure out of $h$ ; that memorisation inflates the real score, so we must subtract it away.
Compute. $Sel = 0.94 - 0.85 = 0.09.$ Why this step? Only $9$ points of the $94$ come from genuine structure in $h$ ; the rest is probe capacity — a near-false-positive result.
Judge and fix. Selectivity of $0.09$ is low: the probe is too expressive and mostly memorising. Shrink the probe (stronger L2, fewer hidden units) until $A_{ctrl}$ falls near the majority/uniform baseline, then re-measure $Sel$ . Only a high selectivity lets you trust the claim "layer $l$ encodes $τ$ ". Pair with Causal Intervention Methods to confirm the info is actually used, not just present. Why this step? Selectivity turns the question "is it there?" into the stronger "is it there and not just memorised?" — the safeguard against the parent note's Mistake 1.

Verify: $0.94 - 0.85 = 0.09$ . ✔

Example 8 — real-world word problem [Cell E1]

Forecast: With 2000 balanced samples over two classes, what's the chance baseline? Is 1740 good?

Steps.

Accuracy. $A = 1740/2000 = 0.87$ . Why this step? Raw score — but by itself meaningless (Cell B1 warning).
Baseline. Balanced binary ⇒ chance $= 1/2 = 0.50$ . Gain over baseline $= 0.87 - 0.50 = 0.37$ . Why this step? Confirms real lift above guessing (this task is binary, so $0.50$ is the correct baseline — contrast Example 6's multi-class case).
Selectivity. $A_{ctrl} = 1020/2000 = 0.51$ ; $Sel = 0.87 - 0.51 = 0.36$ . Why this step? Control sits at chance ⇒ almost all the lift is genuine structure. Verdict: negation is linearly encoded at layer 9, and it's not memorisation.

Verify: $1740/2000 = 0.87$ ; $1020/2000 = 0.51$ ; gain $= 0.37$ ; selectivity $= 0.36$ . ✔

Example 9 — exam twist: probe beats the model [Cell F1]

Forecast: Can the model "know" something it fails to use?

Steps.

Separate presence from usage. Probing measures presence in $h$ ( $92%$ = strongly encoded). Output accuracy measures usage in the final decision ( $65%$ ). Why this step? These are different quantities; $92 \neq = 65$ is not a contradiction.
Interpret the gap. The gap $92 - 65 = 27$ points means the information exists but the model's later layers / decoding fail to exploit it — a wiring problem, not a knowledge problem. Why this step? Points the fix downstream (decoding, fine-tuning the head), not at pre-training.
Caution. Confirm with a causal test — patch the gender direction and see if outputs change (Causal Intervention Methods). A probe alone can't prove the model would use it. Also relevant to Adversarial Robustness: encoded-but-unused features can be adversarially flipped. Why this step? Closes the presence-vs-usage loop rigorously.

Verify: Gap $92 - 65 = 27$ ; presence $>$ usage, no contradiction. ✔

Recall Self-check

A probe hits 0.70 on a 70/30 imbalanced task — is the property encoded? ::: No — that equals the majority baseline; gain over baseline is 0. Linear probe 0.667, feature-map probe 1.0 on the same 6-point set — what does the gap mean? ::: Info is present but nonlinearly entangled (Cell A3). Real accuracy 0.94, control accuracy 0.85 — trust it? ::: Weakly — selectivity is only 0.09, the probe is partly memorising (Cell C2). A point lies exactly on the boundary — what is its signed score and the margin? ::: Signed score 0, so margin γ=0 (Cell A2, knife-edge). For a 17-tag POS probe, what is the uniform-chance baseline? ::: 1/17 ≈ 0.059, not 0.50 — it's multi-class. Probe 0.92 but model output 0.65 — contradiction? ::: No — presence ≠ usage (Cell F1).