Probing classifiers — Hinglish notes

6.3.5 · AI-ML › Interpretability & Explainability

Intuition Core Intuition

Socho ki tumne ek massive neural network train kiya English se French translate karne ke liye. Karte karte, woh internal representations (hidden layer activations) banata hai. Kya usne grammar seekhi? Part-of-speech tags? Syntax trees? Hum sirf weights dekh ke nahi bata sakte. Isliye, hum representations ko probe karte hain—ek simple auxiliary classifier (jaise logistic regression) train karke frozen network activations par, taaki koi specific linguistic property predict ho sake. Agar probe high accuracy achieve kare, toh representation us property ko linearly encode karti hai—woh "andar hai" aur easily extract ho sakti hai.

Yeh kyun important hai: Probing batata hai ki learned representations mein kya information captured hai, kahan (kaun sa layer), aur kitne explicitly (linear vs. nonlinear). Yeh interpretability ke liye ek diagnostic tool hai.

Definition & Problem Setup

Definition Probing Classifier

Ek probing classifier (ya diagnostic classifier) ek supervised model hai—typically ek simple linear classifier ya shallow MLP—jo ek pre-trained model $f_{θ}$ ki frozen representations $h$ se ek auxiliary task $τ$ predict karne ke liye train hota hai.

Setup:

Pre-trained model $f_{θ} : X \to Y$ layer $l$ par intermediate representations $h^{(l)} (x)$ produce karta hai.
Auxiliary dataset $D_{probe} = {(x_{i}, y_{i}^{τ})}$ jisme task $τ$ ke labels hain (jaise part-of-speech tags, syntactic structure, semantic role).
Probe $g_{ϕ} : h^{(l)} \to \overset{y}{^}^{τ}$ train karo frozen $f_{θ}$ ke saath. Probe accuracy $A_{τ}^{(l)}$ measure karo.

Key principle: High $A_{τ}^{(l)}$ ka matlab hai ki layer $l$ ki representations $τ$ ke baare mein information linearly encode karti hain. Low accuracy ka matlab hai ki info absent hai ya complex nonlinear extraction chahiye.

"Frozen" representations kyun? Agar hum $f_{θ}$ ko fine-tune karte, toh model probing ke dauran $τ$ encode karna seekh sakta tha—hum model ki $τ$ seekhne ki capacity measure karte, na ki woh pre-training ke dauran already seekh chuka hai ya nahi.

Derivation: Probe Accuracy Hume Kya Batati Hai?

Intuition ko formalize karte hain. Maano $h^{(l)} \in R^{d}$ layer- $l$ activations hain, aur hum linear classifier se probe karte hain:

\overset{y}{^}^{τ} = σ (w^{⊤} h^{(l)} + b)

jahan $σ$ softmax (multiclass) ya sigmoid (binary) hai, $w \in R^{d}$ , $b \in R$ .

High probe accuracy ka kya matlab hai?

Information theoretically: $h^{(l)}$ mein mutual information $I (h^{(l)}; Y^{τ}) > 0$ hai jahan $Y^{τ}$ task $τ$ ke liye true label distribution hai. High accuracy $⟹$ high MI.
Geometrically: $τ$ ke classes representation space mein linearly separable hain. Decision boundary $w^{⊤} h^{(l)} + b = 0$ space ko cleanly partition karta hai.
Causally: Pre-training objective ne aise features induce kiye jo $τ$ se correlate karte hain. Yeh tab hota hai jab $τ$ main task ke liye ek useful intermediate step ho, ya data structure $τ$ ko encode hone par force kare.

Separability condition ki derivation:

Perfect linear separability ke liye, hume ek hyperplane chahiye jaise ki:

{w^{⊤} h_{i}^{(l)} + b > 0 w^{⊤} h_{i}^{(l)} + b < 0 if y_{i}^{τ} = + 1 if y_{i}^{τ} = - 1

Margin $γ$ define karo:

γ = i min \frac{y _{i}^{τ} ( w ^{⊤} h _{i}^{(l)} + b )}{∥ w ∥}

Agar $γ > 0$ , toh classes linearly separable hain. Probe training (jaise logistic loss se) $w, b$ dhundhti hai jo margin-jaisi quantities maximize kare.

Yeh step kyun? Margin quantify karta hai ki linear separation kitna "easy" hai. Large margin $⟹$ robust linear encoding. Small margin $⟹$ probe mushkil se kaam karta hai, suggesting info nonlinearly encoded ya noisy hai.

Methodology: Probing Study Kaise Run Karein

Formula Probing Protocol (Step-by-Step)

1. Auxiliary tasks ${τ_{1}, τ_{2}, \dots}$ choose karo:

Linguistic: POS tags, dependency parsing, named entities, semantic roles.
Vision: object presence, spatial relations, texture vs. shape.
Goal: aise tasks jo model ne kya seekha iske baare mein hypotheses test karein.

2. Representations extract karo: Har layer $l \in {1, 2, \dots, L}$ aur har sample $x \in D_{probe}$ ke liye: $h^{(l)} (x) = f_{θ}^{(l)} (x)$ Inhe frozen activations ke roop mein store karo.

3. Har layer $l$ aur task $τ$ ke liye probe $g_{ϕ}^{(l)}$ train karo: $min_{ϕ} \frac{1}{N} \sum_{i = 1}^{N} L (\overset{y}{^}_{i}^{τ}, y_{i}^{τ}) + λ R (ϕ)$ jahan $L$ cross-entropy hai, $R (ϕ)$ L2 regularization hai (probe complexity control karta hai).

Regularization kyun? Probe ko idiosyncrasies memorize karne se rokta hai. Hum test karna chahte hain ki info easily accessible hai ya nahi, na ki ek complex probe usse extract kar sakta hai ya nahi.

4. Evaluate karo: Held-out test set par accuracy $A_{τ}^{(l)}$ compute karo. Layers ke across compare karo.

5. Interpret karo:

Layer-wise trends: Early layers low-level features (syntax) encode karte hain, deeper layers semantics encode karte hain.
Task comparisons: High $A_{POS}$ lekin low $A_{semantics}$ $⟹$ syntax explicit hai, semantics entangled hain.

Example calculation: Maano hum BERT ko POS tags ke liye probe karte hain.

Layer 0 (word embeddings): $A_{POS}^{(0)} = 82%$ (word identity ka POS se strong correlation hai).
Layer 6 (middle): $A_{POS}^{(6)} = 97%$ (contextualized representations POS ko trivial bana dete hain).
Layer 12 (output): $A_{POS}^{(12)} = 95%$ (thoda drop; layer downstream task par focus karti hai, syntax par nahi).

Yeh pattern kyun? Middle layers local syntax (MLM ke liye zaroori) aur global context ko balance karte hain. Output layers pre-training objective ke liye specialize hoti hain, potentially task-irrelevant syntax ko discard karti hain.

Worked Example 1: Subject-Verb Agreement ke liye Probing

Task: Kisi sentence se predict karo ki subject aur verb number mein agree karte hain ya nahi (singular/plural).

Setup:

Model: GPT-2 (12 layers).
Dataset: 10k sentences with labels {agree, disagree}.
Probe: Verb token position par activations par Logistic regression.

Step 1: Representations extract karo

Sentence "The cat sleps" ke liye, layers $l = 0, 1, \dots, 11$ par $h_{sleps}^{(l)}$ extract karo.

Step 2: Probe train karo

Layer $l = 6$ ke liye:

w, b min - \frac{1}{N} i = 1 \sum N [y_{i} lo g σ (w^{⊤} h_{i}^{(6)} + b) + (1 - y_{i}) lo g (1 - σ (w^{⊤} h_{i}^{(6)} + b))] + 0.01∥ w ∥^{2}

Yeh step kyun? Cross-entropy loss measure karta hai ki linear projection $w^{⊤} h^{(6)}$ classes ko kitna achha separate karta hai. L2 penalty $0.01∥ w ∥^{2}$ ensure karta hai ki $w$ chhota rahe, yeh test karte hue ki info easily linear hai.

Result: $A_{agreement}^{(6)} = 94%$ .

Interpretation: Layer 6 representations mein ek linear subspace hai jo subject-verb agreement encode karta hai. GPT-2 ne language modeling ke dauran yeh syntactic rule seekha, kyunki verb predict karne ke liye subject number track karna zaroori hai.

Worked Example 2: Semantic Role (Agent vs. Patient) ke liye Probing

Task: "The dog chased the cat" jaisi sentences mein identify karo ki noun agent (doer) hai ya patient (receiver).

Setup:

Model: BERT.
Dataset: 5k sentences with semantic role labels.
Probe: Noun token activations par 2-layer MLP (hidden size 100).

Linear ki jagah MLP kyun? Semantic roles ke liye features ka nonlinear combination zaroori ho sakta hai (jaise position + grammatical relation + verb semantics).

Step 1: $h_{noun}^{(l)}$ extract karo

Step 2: MLP probe train karo

\overset{y}{^} = softmax (W_{2} ReLU (W_{1} h^{(l)} + b_{1}) + b_{2})

Results:

Layer 8: $A_{role}^{(8)} = 78%$ (MLP), $A_{role}^{(8)} = 62%$ (linear probe).
Layer 10: $A_{role}^{(10)} = 89%$ (MLP), $A_{role}^{(10)} = 71%$ (linear).

Interpretation: Semantic roles layer 8 mein present hain lekin nonlinear extraction chahiye (MLP linear se bahut better hai). Layer 10 tak, roles zyada linearly accessible ho jaate hain (gap kam ho jaata hai). Iska matlab hai ki deeper layers semantic structure ko "untangle" karti hain.

Yeh step kyun? Linear vs. nonlinear probes compare karne se pata chalta hai ki info kaise encoded hai. Linearly separable $⟹$ explicit representation. Nonlinearly separable $⟹$ entangled, disentanglement chahiye.

Common Mistakes & Steel-Man Fixes

Diagram: Layer-Wise Probing Accuracy Across Tasks

Figure Explanation: Yeh diagram dikhata hai ki teen tasks (POS tags, dependency parsing, semantic similarity) ke liye probe accuracy ek 12-layer Transformer mein layers ke across kaise vary karti hai. POS tags early peak karte hain (layer 4-6), syntax mid-network peak karta hai (layer 6-8), aur semantics baad mein emerge hoti hain (layer 8-12). Yeh representational hierarchy ko visualize karta hai: lower layers surface form encode karte hain, middle layers structure encode karte hain, upper layers meaning encode karte hain.

Connections to Other Concepts

Representation Learning: Probing measure karta hai ki representations kya encode karti hain.
Attention Mechanisms: Attention weights ko probe karne se pata chalta hai ki kaun se tokens decisions influence karte hain.
Fine-tuning vs. Feature Extraction: Probing feature extraction hai—hum frozen features ko ek naye task ke liye use karte hain.
Causal Intervention Methods: Probing ka complement; interventions usage test karte hain, probing presence test karta hai.
Adversarial Robustness: Adversarial representations ko probe karne se pata chalta hai ki kaun se features robust hain vs. spurious.
Disentangled Representations: High linear probe accuracy $⟹$ feature disentangled hai (ek independent subspace occupy karta hai).

Hidden Feynman Recall Block

Recall Explain Like I'm 12

Socho tumne ek robot ko English se French translate karna sikhaya. Robot ka ek "brain" (neural network) hai jisme bahut saari layers hain, jaise ek building mein floors. Tum sochte ho: Kya robot grammar samajhta hai? Kya woh jaanta hai ki kaun se words nouns hain vs. verbs?

Tum robot ka brain open karke padh nahi sakte—woh bahut complicated hai. Isliye, tum ek game khelate ho: Tum robot ko kuch sentences dikhate ho aur poochte ho, "Mujhe har word ka part-of-speech tag batao." Lekin yahan ek trick hai—tum robot ka brain freeze kar dete ho (use kuch naya nahi seekhne dete) aur sirf ek chhota helper ("probe") train karte ho jo robot ke already seekhe hue cheez ke basis par tags guess kare.

Agar helper 95% baar correctly guess kare, toh robot ke brain mein already part-of-speech info hai, aisa store ki easily extract ho sake. Agar helper sirf 60% guess kare, toh info ya toh missing hai ya complicated tarike se chhipi hui hai.

Yeh kyun matter karta hai? Yeh batata hai ki robot ne kya seekha aur kahan (kaun si layer mein). Shayad pehli kuch layers spelling jaanti hain, middle layers grammar jaanti hain, aur last layers meaning jaanti hain. Yeh hume robot par trust karne aur uski galtiyan theek karne mein help karta hai.

Mnemonic

Active Recall Flashcards

#flashcards/ai-ml

Probing classifier kya hota hai? :: Ek simple supervised model (jaise logistic regression) jo frozen pre-trained representations par train hota hai ek auxiliary task predict karne ke liye, yeh measure karta hai ki representation task-relevant information linearly encode karti hai ya nahi.

Probing ke dauran hum pre-trained model ko freeze kyun karte hain?

Yeh test karne ke liye ki model ne pre-training ke dauran already kya seekha, na ki woh kya seekh sakta hai. Unfreeze karne par model probe task encode karne ke liye adapt ho sakta tha, measurement confound ho jaata.

High probe accuracy hume kya batati hai?

Representation mein auxiliary task ke baare mein linearly accessible form mein information hai (geometrically: classes linearly separable hain; information-theoretically: high mutual information hai).

Linear probe 65% accuracy deta hai, MLP probe 88% deta hai. Iska kya matlab hai?

Information present hai lekin entangled, nonlinear tarike se encoded hai. Representation ko feature extract karne ke liye complex transformations chahiye.

Linear separability mein margin kya hai?

γ = min_{i} \frac{y _{i} ( w ^{⊤} h _{i} + b )}{∥ w ∥}

, kisi bhi point se decision boundary ki minimum distance. Positive margin linear separability imply karta hai.

Probe training ke dauran regularize kyun karte hain?

Overfitting rokne ke liye aur yeh ensure karne ke liye ki hum easy accessibility test kar rahe hain. Hum jaanna chahte hain ki info explicitly encoded hai ya nahi, na ki ek complex probe dataset memorize kar sakta hai ya nahi.

Probing dikhata hai ki layer 6 POS tags 95% accuracy par encode karta hai. Kya model apne task ke liye POS tags use karta hai?

Zaroor nahi. Probing presence measure karta hai, usage nahi. Model POS tags byproduct ke roop mein encode kar sakta hai lekin causally unpar depend nahi karta. Causal interventions se test karo.

Transformers mein linguistic probes ke liye typical layer-wise pattern kya hai?

Early layers surface features (POS, morphology) encode karte hain, middle layers syntax (dependencies, phrase structure) encode karte hain, late layers semantics (entailment, paraphrase) encode karte hain. Yeh representational hierarchy hai.

Probe accuracy ke liye baseline kaise establish karte hain?

Random representations (jaise Gaussian noise) ya shuffled labels probe karo. Agar model ki probe accuracy is baseline se zyada nahi, toh feature encoded nahi hai.

Probing aur causal intervention mein kya fark hai?

Probing measure karta hai ki koi feature representations mein present hai ya nahi. Causal intervention (jaise ablation) test karta hai ki feature task ke liye use hota hai ya nahi. Complete interpretability ke liye dono zaroori hain.