Intuition The one core ideaA big neural network, while learning to do its main job (say, translate or predict the next word), quietly builds up internal "notes" about the input — numbers stored in its hidden layers. Probing means freezing those notes and training a tiny separate classifier to guess a specific property (like "is this word a noun?"): if the tiny classifier succeeds, the property was already written into the notes.
This page assumes you have seen none of the notation on the parent page . We build every symbol from a picture. Read top to bottom — each item uses only the ones above it.
x X x one single example we feed the network — one sentence, one image. The fancy X x ∈ X x
Picture it: a box labelled inputs holding sentences; you reach in and pull out one — that pulled-out sentence is x
Y Y set of all things the model was trained to output — the collection of possible French sentences, or the set of next-word probabilities. Just as X Y f θ : X → Y f θ X Y
Do not confuse this with the task labels y τ Y own main-job output; y τ property we later probe for.
f θ f θ already-trained network . The little θ θ
Look at the figure. The input x Y middle .
Intuition Why we look inside instead of at the outputThe output only tells us the final answer. The interesting question — did it secretly learn grammar? — lives in the intermediate stages, not the final print-out.
This is the single most important symbol on the whole parent page, so we build it slowly.
Definition Representationh ( l ) x layer transforms it into a list of numbers. That list is the layer's representation of the input, written h vector (a whole list, not one number). The superscript ( l ) which layer produced it: h ( 1 ) h ( 6 )
h l l = 0 l = L L This upright L L
The figure shows each layer emitting its own arrow of numbers. Why does the topic need this symbol? Because probing asks where information lives — and "where" means "at which l
d R d h ( l ) h ( l ) ∈ R d h d R R d d
Picture it: if d = 2 h ( x -coord , y -coord ) d = 3 d = 768 idea — a single point positioned by its coordinates — is exactly the same.
Frozen means we do not change θ h ( l )
Common mistake Why not just fine-tune?If we let θ learn the property on the spot to make the probe look good. Then we'd be measuring "could it learn this?" instead of "did it already know this?" Freezing forces an honest answer.
See Fine-tuning vs. Feature Extraction for the full contrast — probing is the extreme "feature extraction" end where nothing in the base model moves.
τ y τ τ specific property we are testing for — e.g. "part-of-speech" or "singular vs. plural." For each input x i y i τ label for task τ i i = 1 , 2 , … , N N
The collection of pairs is the probe dataset : D probe = {( x i , y i τ )}
Picture it: a spreadsheet. Left column = sentences x i y i τ NOUN, VERB).
Definition From categories to signed numbersProperties like POS come as words (NOUN, VERB), but the boundary picture in §5 needs numbers . For a two-class task we pick a convention: one class becomes + 1 − 1 agree ↦ + 1 disagree ↦ − 1 ↦
For a task with more than two categories (POS has dozens), we do not use one signed number; instead each category gets its own output slot and the probe uses softmax over slots. The tidy ± 1 two-class special case — the cleanest one to draw and reason about.
g ϕ g ϕ small classifier we train — a fresh, simple model that takes a representation h ( l ) ϕ separate from the frozen θ ϕ
The simplest probe is a linear classifier . Here is every piece of its formula:
y ^ τ = σ ( w ⊤ h ( l ) + b )
Definition Breaking the formula apart
w weight vector , same length d h w ⊤ h ( l ) dot product : multiply matching entries and add them all up, giving one single number. (The little ⊤ b bias , one number that shifts the result up or down.σ squashing function that turns the raw number into a probability between 0 and 1 (sigmoid) or into class-probabilities (softmax).y ^ τ prediction (the hat "^ y τ
Why a dot product? Because it is the mathematical way to ask "how much of direction w h One number that measures alignment — exactly what we need to sort points into two groups.
The figure shows dots (representations) in a 2-D space, with the line w ⊤ h + b = 0 ± 1 everything on one side gets label + 1 − 1 That line is the decision boundary .
Definition Linear separabilityIf a straight boundary can cleanly split the two classes, they are linearly separable — the property is written into h not cleanly linear there — the info may be entangled. That connects directly to Disentangled Representations .
γ margin γ γ γ
Caution — read this carefully. The margin is the native language of a max-margin classifier (an SVM), which explicitly maximises γ cross-entropy (§6) does not directly maximise γ γ intuition-only ruler for "how cleanly separated are the classes?", not as the quantity the cross-entropy probe optimises. When the parent page mentions a margin, treat it as this diagnostic measure of separation quality, not as the objective of the logistic probe.
L L penalty score : big when the probe's guesses are wrong, small when right. Training means turning the ϕ L cross-entropy loss, which specifically punishes confident wrong answers hard. Reminder: this script L L
A τ ( l ) A τ ( l ) fraction of held-out examples the probe gets right , for task τ l This is the number the whole study reports. High A τ ( l ) l
Definition Regularizationλ R ( ϕ ) R ( ϕ ) ∥ w ∥ 2 w extra penalty on probe complexity ; λ simple , so a high score means the info was easy to extract, not that a clever probe dug it out.
Definition The label as a random variableY τ y i τ one observed answer for a specific example x i whole statistical behaviour across all examples — its spread, its uncertainty — we use the capital Y τ random variable : the label thought of as "a value that varies from example to example, with some distribution." So y i τ Y τ
Definition Mutual informationI ( h ( l ) ; Y τ ) how much knowing the representation h ( l ) Y τ . Zero means the representation tells you nothing about the property; large means it tells you a lot. High probe accuracy implies high mutual information. (Note the capital Y τ distribution of labels, so it must use the random variable, not a single observed y i τ
Common mistake Presence is not usageA high A τ ( l ) there , not that the model acts on it . To test whether the model actually uses a feature you must intervene — see Causal Intervention Methods . Probes on their own are a microscope, not a wrench.
The representations we probe often come from Attention Mechanisms inside transformers, and a probe that succeeds on tiny perturbed inputs but fails on natural ones is a warning sign studied under Adversarial Robustness .
Model f theta stacked layers
Hidden representation h at layer l
Vector in d dimensional space
Dot product w dot h plus b
Decision boundary and separation
Task tau labels y mapped to plus or minus one
Probe g phi trained on frozen h
Interpretation which layer encodes what
What does h ( l ) The list of numbers (a vector) that layer l
Why must the base model be frozen during probing? So the probe measures what the model already learned, not what it can learn on the spot.
What does the dot product w ⊤ h How much of direction w h
What is θ ϕ θ ϕ
How does a categorical label like agree/disagree become a number for the boundary picture? By a chosen convention, e.g. agree ↦ + 1 disagree ↦ − 1
Does a cross-entropy-trained probe maximise the margin γ No — γ γ
What is the difference between y i τ Y τ y i τ i Y τ
How do the two look-alike symbols L L Upright L L
What does high A τ ( l ) It tells you the property is present and linearly readable at layer l
