Probing classifiers: Exercises — AI-ML deep dive

Intuition How to use this page

Each problem below is a rung on a ladder. L1 just checks you recognise the vocabulary; by L5 you are designing a probing study from scratch and defending it. Try each problem with pen and paper before opening its Solution callout. If a level's [!mistake] box describes exactly the wrong turn you took — good, that is the whole point. This page tests the parent note Probing classifiers; when a concept from a neighbour topic appears, we link it.

What a "probe" is, from zero. Freeze a trained network. Pick one layer $l$ . Feed it inputs and read off the numbers it produces — that vector is the representation $h^{(l)}$ . Now train a tiny separate model (the probe) whose only job is to guess some property from that vector. If the tiny model succeeds, the property was already "written into" $h^{(l)}$ . That is all probing is: reading, never rewriting.

Definition Notation used on this page (read first)

$τ$ (Greek "tau") — the auxiliary task we probe for: the specific property we ask the probe to predict (e.g. part-of-speech tag, subject–verb agreement, semantic role). It is just a name for "which question the probe is answering".
$A_{τ}^{(l)}$ — the probe accuracy: on a held-out test set, the fraction of examples the probe labels correctly when reading layer- $l$ representations for task $τ$ , written as a percent. Formally, if the probe makes $N$ predictions and $c$ are right, $A_{τ}^{(l)} = \frac{c}{N} \times 100%$ . When the task is clear we drop the subscript and write $A^{(l)}$ .
MLP = Multi-Layer Perceptron: a small neural network with at least one hidden layer and a nonlinearity, e.g. $\overset{y}{^} = softmax (W_{2} ReLU (W_{1} h + b_{1}) + b_{2})$ . Because of the ReLU it can carve curved decision regions, unlike a linear probe which only draws one flat hyperplane. We call it the "nonlinear probe".

Level 1 — Recognition

Can you name the parts?

Exercise 1.1

State, in one sentence each, what these three symbols mean: $f_{θ}$ , $h^{(l)}$ , $g_{ϕ}$ .

Recall Solution

$f_{θ}$ — the big pre-trained model we are studying (weights $θ$ , kept frozen). It maps an input $x$ to some output, but we care about its insides.
$h^{(l)}$ — the representation: the vector of activations that $f_{θ}$ produces at layer $l$ for a given input. This is the only thing the probe is allowed to look at.
$g_{ϕ}$ — the probe: a small, separately trained classifier (weights $ϕ$ ) that takes $h^{(l)}$ and predicts an auxiliary label.

Exercise 1.2

True or false: "When we train the probe, we also update the weights $θ$ of the big model." Justify.

Recall Solution

False. The big model is frozen. Only $ϕ$ (the probe's weights) change. If $θ$ changed, the model could learn the property during probing, and we'd no longer be measuring what it knew beforehand — we'd be measuring its capacity to learn. Compare Fine-tuning vs. Feature Extraction: fine-tuning unfreezes; probing does not.

Exercise 1.3

Recall from the notation box that $τ$ names the auxiliary task and $A_{τ}^{(l)}$ is the probe's test accuracy for task $τ$ at layer $l$ . Match each accuracy claim to its meaning: (a) $A_{τ}^{(l)} = 95%$ (b) $A_{τ}^{(l)} = 50%$ on a balanced 2-class task.

Recall Solution

(a) The property $τ$ is strongly and linearly present in layer $l$ — a simple hyperplane separates the classes cleanly, so $95%$ of held-out examples are labelled correctly.
(b) $50%$ on a balanced binary task equals random guessing, so the probe found nothing — the info is absent, or so entangled a linear probe cannot reach it.

Level 2 — Application

Plug into the formulas.

Exercise 2.1

A binary linear probe outputs $\overset{p}{^} = σ (w^{⊤} h + b)$ where $σ (z) = \frac{1}{1 + e ^{- z}}$ is the sigmoid (it squashes any real number into $(0, 1)$ , so we can read it as a probability). Labelling convention: we call the output the positive class ( $y = + 1$ ) when $\overset{p}{^} > 0.5$ , equivalently when $z > 0$ ; otherwise the negative class ( $y = - 1$ ). Given $w = (2, - 1)$ , $b = 0.5$ , and $h = (1, 3)$ , compute the pre-activation $z = w^{⊤} h + b$ and the predicted probability $\overset{p}{^}$ . Which class?

Recall Solution

The dot product $w^{⊤} h$ multiplies matching entries and sums them: $z = (2) (1) + (- 1) (3) + 0.5 = 2 - 3 + 0.5 = - 0.5.$ Then $\overset{p}{^} = σ (- 0.5) = \frac{1}{1 + e ^{0.5}} \approx 0.3775.$ Since $\overset{p}{^} < 0.5$ (equivalently $z < 0$ ), by our convention the probe predicts the negative class $y = - 1$ .

Exercise 2.2

Two data points lie on either side of a probe's boundary: $h_{1} = (1, 0)$ with label $y_{1} = + 1$ , and $h_{2} = (- 1, 0)$ with label $y_{2} = - 1$ . The probe is $w = (1, 0)$ , $b = 0$ . Compute the margin $γ = min_{i} \frac{y _{i} ( w ^{⊤} h _{i} + b )}{∥ w ∥} .$ (The margin is the signed distance from the closest point to the boundary; positive means correctly separated.)

Recall Solution

Why divide by $∥ w ∥$ ? The plain quantity $w^{⊤} h_{i} + b$ measures "how far past the boundary" but in units that stretch if you scale $w$ up. Dividing by $∥ w ∥$ (the length of $w$ ) cancels that scaling and converts the number into an honest geometric distance in the representation space — the perpendicular distance from the point to the hyperplane. Multiplying by $y_{i}$ makes the distance positive exactly when the point is on its correct side. $∥ w ∥ = 1^{2} + 0^{2} = 1$ .

Point 1: $y_{1} (w^{⊤} h_{1} + b) = (+ 1) (1) = 1$ , distance $1/1 = 1$ .
Point 2: $y_{2} (w^{⊤} h_{2} + b) = (- 1) (- 1) = 1$ , distance $1/1 = 1$ . $γ = min (1, 1) = 1.$ Positive margin ⇒ the two classes are linearly separable by this probe, and comfortably so.

Exercise 2.3

A probe's L2-regularised objective is $J (w) = cross-entropy loss L (w) + λ ∥ w ∥^{2}, λ = 0.01,$ where the loss is the binary cross-entropy $L (w) = - \frac{1}{N} \sum_{i} [y_{i} lo g \overset{p}{^}_{i} + (1 - y_{i}) lo g (1 - \overset{p}{^}_{i})]$ (it is small when the probe's probabilities $\overset{p}{^}_{i}$ match the true labels). If $w = (3, 4)$ , compute the penalty term $λ ∥ w ∥^{2}$ . Explain in one line why we add it.

Recall Solution

$∥ w ∥^{2} = 3^{2} + 4^{2} = 25$ , so penalty $= 0.01 \times 25 = 0.25$ . Why: the cross-entropy term rewards fitting the labels; the penalty keeps $w$ small so the probe stays simple. We want to test whether the info is easily linearly accessible — not whether a huge, memorising probe can dig it out.

Level 3 — Analysis

Read patterns and reason about them.

Exercise 3.1

A BERT POS-tag probing sweep gives: $A^{(0)} = 82%$ , $A^{(6)} = 97%$ , $A^{(12)} = 95%$ . Sketch the curve mentally and explain the shape: why does accuracy rise then dip? Reference the figure.

Figure — the horizontal axis is the layer index $l$ (0, 6, 12); the vertical axis is POS probe accuracy $A^{(l)}$ in percent. The teal curve climbs steeply from $82%$ at layer 0 to a peak of $97%$ at layer 6, then dips slightly to $95%$ at layer 12. A dashed line at $50%$ marks chance for a balanced binary task. The shape is an asymmetric "hump": a strong rise, a small fall.

Recall Solution

Rise (0→6): layer 0 is raw word embeddings — POS is guessable from word identity alone ( $82%$ ), but ambiguous words ("run" noun vs verb) cap it. Middle layers mix in context (Attention Mechanisms let each token read its neighbours), so POS becomes almost trivial ( $97%$ ).
Dip (6→12): the top layer specialises for the pre-training objective (masked-language modelling), spending capacity on task-specific prediction and slightly discarding generic syntax — hence $95%$ . The takeaway: syntactic info peaks in the middle, not at the end.

Exercise 3.2

For semantic roles, layer 8 gives $A_{MLP} = 78%$ (nonlinear probe) but $A_{linear} = 62%$ (linear probe). Compute the gap. What does a large linear-vs-nonlinear gap tell you about how the info is stored?

Recall Solution

Gap $= 78 - 62 = 16$ percentage points. A large gap means: the info is present (the MLP, which can bend its decision boundary, finds it) but is not linearly separable — it needs nonlinear combinations of features that only a hidden layer can form. In representation terms the classes are entangled; a straight hyperplane can't split them. Cleanly separating such factors is the goal of Disentangled Representations.

Exercise 3.3

At layer 10 the gap shrinks: $A_{MLP} = 89%$ , $A_{linear} = 71%$ (gap $= 18$ ). A student says "the gap grew, so layer 10 entangles roles more than layer 8." Is that right?

Recall Solution

No — you must compare the right quantity. The absolute gap ( $16 \to 18$ ) is a red herring because the ceiling rose too. Look at the linear probe alone: it jumped $62% \to 71%$ , meaning roles became more linearly accessible at layer 10. Deeper layers untangle structure; the linear probe's rise is the honest signal, not the raw gap.

Level 4 — Synthesis

Combine ideas into a design.

Exercise 4.1

You suspect a vision model encodes shape but not texture. Design a two-probe experiment: name the tasks, the labels, what "shape encoded, texture not" would look like in the numbers.

Recall Solution

Task A (shape): classify object category on images where texture is scrambled but silhouette intact.
Task B (texture): classify texture patch label independent of shape. Freeze the model, extract $h^{(l)}$ per layer, train a linear probe for each task. "Shape yes, texture no" shows up as $A_{shape}^{(l)}$ high (say $> 85%$ ) while $A_{texture}^{(l)} \approx$ chance across layers. Run it per layer to see where shape emerges.

Exercise 4.2

Design a control that rules out the boring explanation "the probe itself is powerful enough to learn anything." Describe the control task and the number you'd compare against.

Recall Solution

Use a control task: keep the same inputs, but replace the true labels with random ones. Concretely, you assign each word type a fixed random label once (so "cat" always gets, say, label 3, "dog" always label 1 — the assignment is arbitrary but consistent, which is what "fixed per word type" means). There is now no real linguistic structure for the probe to exploit; it can only win by memorising the arbitrary word→label table. Train the same probe on this control and report selectivity $= A_{real} - A_{control}$ .

If the real task scores $95%$ but the random-label control also scores $90%$ , selectivity is tiny — the probe is just memorising, and the high accuracy is not evidence the representation encodes $τ$ .
High selectivity (real ≫ control) means the representation, not the probe's raw power, carries the signal.

Level 5 — Mastery

Full reasoning, edge cases, defence.

Exercise 5.1

A probe reaches $50%$ on a balanced binary agreement task at layer 3, but $94%$ at layer 6. A colleague concludes "the model does not encode agreement at layer 3." Steel-man and then critique this conclusion, covering the degenerate cases.

Recall Solution

Steel-man: a linear probe at $50%$ = chance, so no linearly-accessible agreement signal at layer 3. That much is fair. Critique — the info might still be there, just entangled:

Try a nonlinear (MLP) probe at layer 3. If it jumps to, say, $80%$ , agreement is present but nonlinearly encoded — the linear probe was blind to it.
Check the control task — maybe $50%$ is genuinely floor, confirming absence.
Degenerate case: if both linear and MLP sit at chance, and the control does too, then agreement really is absent at layer 3. Correct conclusion: "no linearly accessible agreement at layer 3" — silence from a linear probe is not proof of absence.

Exercise 5.2

An adversary slightly perturbs inputs so a fixed probe's accuracy collapses from $94%$ to $55%$ , while the model's own outputs barely change. What does this reveal, and which neighbouring topic explains it?

Recall Solution

The probe was reading a fragile, non-robust direction in representation space. Tiny input changes moved $h^{(l)}$ across the probe's hyperplane without changing the model's task behaviour — so the "encoding" the probe found was partly a brittle correlation, not a stable, causally-used feature. This is the domain of Adversarial Robustness: a feature that survives perturbation is trustworthy; one that doesn't may be a spurious probe artefact.

Exercise 5.3

Full numeric synthesis. A linear probe has learned $w = (1, - 2, 2)$ , $b = - 1$ , and uses the convention class $+ 1$ iff $z > 0$ . Three verb-token representations arrive: $h_{A} = (2, 1, 1)$ , $h_{B} = (0, 0, 0)$ , $h_{C} = (1, 1, 3)$ . (i) Compute each $z = w^{⊤} h + b$ . (ii) Predict class with threshold $z = 0$ . (iii) For the zero vector $h_{B}$ , explain what the prediction depends on and why that is a meaningful edge case.

Recall Solution

(i) Dot products then add $b = - 1$ :

$z_{A} = (1) (2) + (- 2) (1) + (2) (1) - 1 = 2 - 2 + 2 - 1 = 1.$
$z_{B} = 0 + 0 + 0 - 1 = - 1.$
$z_{C} = (1) (1) + (- 2) (1) + (2) (3) - 1 = 1 - 2 + 6 - 1 = 4.$

(ii) With the convention class $+ 1$ iff $z > 0$ : $z_{A} = 1 > 0 \Rightarrow + 1$ ; $z_{B} = - 1 < 0 \Rightarrow - 1$ ; $z_{C} = 4 > 0 \Rightarrow + 1$ .

(iii) Zero-vector edge case: when $h = 0$ , the weight vector contributes nothing, so the prediction is decided entirely by the bias $b$ . Here $z = b = - 1$ , negative. This is meaningful: it tells you the probe's default vote when the representation is empty/uninformative. If a real input ever produced a near-zero $h$ , the probe would just fall back on $b$ — a reminder that the bias, not the data, breaks ties.

Recall One-line self-test

What is the single most important thing probing does not tell you? ::: Whether the model uses the feature — probing shows presence, not usage; for usage you need causal intervention.