Confusion matrix interpretation

2.6.6 · AI-ML › Model Evaluation & Selection

Ise ek medical test result breakdown ki tarah socho: sirf "90% accurate" nahi, balki "90% beemar logon ko correctly identify kiya (sensitivity), 95% healthy logon ko correctly identify kiya (specificity), 10% false alarms, 5% missed cases." Matrix aapko errors mein asymmetry dekhne par majboor karta hai—cancer case miss karna aur false alarm dena ek jaisi baat nahi hai.

What It Is

• True Positive (TP): Correctly predicted positive
• False Negative (FN): Missed positive (Type II error)
• False Positive (FP): False alarm (Type I error)
• True Negative (TN): Correctly predicted negative

YE STRUCTURE KYO? Rows = reality, columns = predictions. Entry $(i,j)$ = "true class $i$, predicted as class $j$." Perfect classifier → diagonal matrix (saare off-diagonal = 0).

Deriving Core Metrics from First Principles

Har metric ek ratio of cell counts hai, jo ek specific sawaal ka jawaab dene ke liye design ki gayi hai.

Accuracy

KYO? "Saari predictions mein se kitni fraction sahi thi?" Diagonal (correct) ko total se divide karo.

YE KAB FAIL HOTA HAI: Imbalanced classes. Agar 95% emails spam nahi hain, toh sabke liye "not spam" predict karne se 95% accuracy milti hai, lekin saara spam miss ho jaata hai. Accuracy class importance ke liye blind hai.

Precision (Positive Predictive Value)

DERIVATION: Tumne positive $(\text{TP}+\text{FP})$ baar predict kiya. Kitne actually positive the? $\frac{\text{TP}}{\text{TP}+\text{FP}}$.

JAWAAB DETA HAI: "Jab main 'positive' kehta hoon, kitni baar main sahi hota hoon?" Positive predictions ki trustworthiness measure karta hai.

YE KYO MATTER KARTA HAI: Spam filter—agar precision low hai, toh kai legitimate emails spam mark ho jaati hain (user frustration). Medical screening—low precision matlab kai healthy logon ko invasive follow-up tests se guzarna padta hai.

Recall (Sensitivity, True Positive Rate)

DERIVATION: $(\text{TP}+\text{FN})$ actual positives hain. Tumne kitne pakde? $\frac{\text{TP}}{\text{TP}+\text{FN}}$.

JAWAAB DETA HAI: "Saare positives mein se kitne maine dhundhe?" Detection ki completeness measure karta hai.

YE KYO MATTER KARTA HAI: Fraud detection—high recall matlab kam frauds slip through hote hain. Cancer screening—high recall matlab kam cases miss hote hain (life-threatening conditions ke liye critical).

Specificity (True Negative Rate)

DERIVATION: $(\text{TN}+\text{FP})$ actual negatives hain. Tumne kitne correctly identify kiye? $\frac{\text{TN}}{\text{TN}+\text{FP}}$.

JAWAAB DETA HAI: "Saare negatives mein se kitne maine correctly reject kiye?" False alarms se bachne ki ability measure karta hai.

RECALL KO COMPLEMENT KARTA HAI: High recall lekin low specificity = "crying wolf" (tum sab pakad lete ho lekin bahut saare false positives flag kar dete ho).

F1-Score

HARMONIC MEAN KYO? Arithmetic mean $\frac{P+R}{2}$ bahut forgiving hai. Agar $P=100\%,R=10\%$, toh arithmetic mean = 55% (misleading). Harmonic mean = 18% (imbalance ko penalize karta hai).

DOOSRI FORM KI DERIVATION: $F_1=\frac{2}{\frac{1}{P}+\frac{1}{R}}=\frac{2}{\frac{\text{TP}+\text{FP}}{\text{TP}}+\frac{\text{TP}+\text{FN}}{\text{TP}}}=\frac{2\text{TP}}{\text{TP}+\text{FP}+\text{TP}+\text{FN}}=\frac{2\text{TP}}{2\text{TP}+\text{FP}+\text{FN}}$

YE STEP KYO? Harmonic mean definition: $\frac{2}{\frac{1}{a}+\frac{1}{b}}$. Precision aur recall substitute karo, denominators simplify karo, TP se multiply karo.

Multi-Class Confusion Matrix

$C$ classes ke liye, matrix $C\times C$ hai. Entry $M[i,j]$ = kitni baar true class $i$ ko class $j$ predict kiya gaya.

Isse padhna:

• Diagonal (85, 90, 92) = correct predictions
• Off-diagonal = confusion patterns
• Row 1 (cat): 10 cats ko dogs samjha, 5 ko birds
• Column 1: 8 dogs, 3 birds ko cats ki tarah misclassify kiya

PER-CLASS METRICS: Class "Cat" ke liye ($i=1$):

• $T{P}_{\text{cat}}=85$ (diagonal)
• $F{N}_{\text{cat}}=10+5=15$ (row 1, off-diagonal)
• $F{P}_{\text{cat}}=8+3=11$ (column 1, off-diagonal)
• $T{N}_{\text{cat}}=(90+2+5+92)=189$ (saare entries jo row/column 1 mein nahi hain)

${\text{Precision}}_{\text{cat}}=\frac{85}{85+11}=\frac{85}{96}=0.885$ ${\text{Recall}}_{\text{cat}}=\frac{85}{85+15}=\frac{85}{100}=0.85$

YE STEP KYO? Class $i$ ke liye, $TP$ diagonal entry hai, $FN$ row sum minus diagonal hai, $FP$ column sum minus diagonal hai, $TN$ baaki sab kuch hai.

MACRO vs MICRO AVERAGING:

• Macro: Har class ke liye metric compute karo, phir average karo (saari classes ko equally treat karta hai)
• Micro: Saari classes ke TP, FP, FN sum karo, ek baar compute karo (class size se weight karta hai)

Worked Example: Medical Diagnosis

Model predictions:

• 80 sick correctly identified (TP)
• 20 sick missed (FN)
• 50 healthy ko sick flag kiya (FP)
• 850 healthy correctly cleared (TN)

Confusion matrix:

Metrics:

$\text{Accuracy}=\frac{80+850}{1000}=0.93\text{ (93\%)}$

YE STEP KYO? Diagonal sum karo (correct), total se divide karo. Achha lagta hai, lekin…

$\text{Precision}=\frac{80}{80+50}=\frac{80}{130}=0.615\text{ (61.5\%)}$

YE STEP KYO? 130 "sick" predictions mein se sirf 80 sahi the. 61.5% trustworthiness—agar tum positive test karte ho, toh sirf ~62% chance hai ki tum actually sick ho.

$\text{Recall (Sensitivity)}=\frac{80}{80+20}=\frac{80}{100}=0.80\text{ (80\%)}$

YE STEP KYO? 100 actual sick mein se 80 pakde. 20% cases miss ho gaye—20 beemar log ghar bhej diye!

$\text{Specificity}=\frac{850}{850+50}=\frac{850}{900}=0.944\text{ (94.4\%)}$

YE STEP KYO? 900 healthy mein se 850 correctly clear kiye. 5.6% false alarm rate.

TRADEOFF: High specificity (kam false alarms), lekin moderate recall (20% sick miss ho gaye). Medicine mein, tum likely threshold lower karoge taaki zyada sick pakad sako (recall boost ho), aur zyada false alarms accept karoge (precision/specificity lower ho).

Common Mistakes

Ye sahi kyun lagta hai: 95% correct bahut achha lagta hai.

Trap: Accuracy class-blind hai. 100 samples mein, 95 negative, 5 positive—"sab negative" predict karna = 95% accuracy, lekin 0% recall (har positive miss ho gaya).

Example: Fraud detection mein 1% fraud rate. "Koi fraud nahi" har jagah = 99% accuracy, lekin bekar (zero fraud pakda).

FIX: Hamesha precision/recall/F1 check karo, especially imbalanced data ke saath. Accuracy akele tabhi meaningful hai jab classes balanced hoon.

Ye sahi kyun lagta hai: Dono TP use karte hain, dono percentages hain, mix up karna aasaan hai.

Trap:

• Precision = "Meri positive predictions mein se kitni sahi thi?" (column view)
• Recall = "Actual positives mein se kitne maine dhundhe?" (row view)

Mnemonic: Precision = Predicted positives (denominator). Recall = Real positives (denominator).

Example: Search engine—high precision = top results relevant hain (kam junk). High recall = zyada relevant docs mile (kam missed). Dono chahiye.

FIX: Denominator pe anchor karo. Precision = $\frac{\text{TP}}{\text{predicted pos}}$, Recall = $\frac{\text{TP}}{\text{actual pos}}$.

Ye sahi kyun lagta hai: Summary metrics (F1, accuracy) report karna aasaan hai.

Trap: Tum kho dete ho ki model kahan fail karta hai. Ho sakta hai model "cat" ko "dog" se confuse karta ho lekin "bird" se kabhi nahi—ye pattern ek single F1 score mein invisible hai.

Example: Multi-class classifier jisme F1 = 0.85. Confusion matrix reveal karta hai: A↔B kabhi confuse nahi karta, lekin C↔D hamesha confuse karta hai. Tum C/D discrimination pe retraining focus karoge, lekin F1 akele ye nahi batata.

FIX: Multi-class ke liye hamesha matrix visualize karo. Systematic confusions dhundho (similar classes, data leakage patterns).

The Precision-Recall Tradeoff

KYO? Lower threshold = zyada liberally positive predict karna. Tum zyada true positives pakdoge (↑ recall), lekin zyada negatives ko bhi positive flag karoge (↓ precision).

Curve: Saare thresholds pe precision vs recall plot karo Precision-Recall curve. Area under curve (AUC-PR) = thresholds ke across model quality ka single-number summary.

Feynman Explanation

• True Positive (TP): Tumne kisi ko doobte dekha aur bachaya. ✓
• False Negative (FN): Koi doob raha tha, lekin tum miss kar gaye. ✗ (sabse bura error—koi mar gaya)
• False Positive (FP): Tum "bachane" ke liye kooде lekin woh bas swimming kar raha tha. ✗ (sharmindagi, time waste)
• True Negative (TN): Banda safely swimming tha, tumne usse akela chhod diya. ✓

Accuracy = "Kitne % time main sahi tha?" (TP + TN) / total. Lekin agar 100 swimmers mein sirf 1 doob raha hai, tum bas baithke kuch nahi karo, aur 99% "accurate" raho jab ki sab doob jaate hain. Bekar.

Recall = "Jo log actually doob rahe the, unme se kitno ko maine bachaya?" Agar recall 70% hai, tum 30% drowning logon ko miss kar gaye. Ye bura hai.

Precision = "Jab main kisi ko bachane ke liye kooда, kitni baar woh actually doob raha tha?" Agar precision 50% hai, aadhe rescues false alarms the (tum safe swimmers pe energy waste kar rahe ho).

Tradeoff: Agar tum paranoid ho aur har baar jab koi splash kare to kood jaate ho (low threshold), tum sabko bachaoге (high recall) lekin bahut saare safe swimmers ko irritate karoge (low precision). Agar tum cautious ho aur sirf obvious drowning ke liye koodte ho (high threshold), kam false alarms (high precision) lekin subtle cases miss ho jaate hain (low recall).

Confusion matrix tumhe dikhata hai ki exactly kitne har type ke mistakes tumne kiye. Phir tum decide karo: lifeguarding mein, drowning person miss karna false alarm se bura hai, isliye tum high recall ke liye optimize karoge (threshold lower karo), chahe precision drop ho jaaye.

Mnemonic

Confusion matrix corners (clockwise from top-left): "TP, FP, TN, FN" → "True, False, True, False" + "Positive, Positive, Negative, Negative"

Ya: "Two Pints, Two Notes" (TP, FP, TN, FN) agar tumhe beer aur music pasand hai.

Connections

• Precision and Recall – detailed derivations, F-beta score
• ROC Curve and AUC – threshold-based models ke liye alternative evaluation
• Class Imbalance Handling – accuracy kyun fail hoti hai, kaise rebalance karein
• Multi-class Classification Metrics – macro/micro averaging, per-class analysis
• Cost-Sensitive Learning – FP vs FN ko alag-alag costs assign karna
• Model Selection Criteria – alag confusion matrices wale models mein se kaise choose karein
• Binary Classification Thresholds – business goals ke liye optimize karne hetu threshold move karna

#flashcards/ai-ml

Binary confusion matrix ke chaar cells kya hain? :: True Positive (TP), False Negative (FN), False Positive (FP), True Negative (TN)

Concept Map