Confusion matrix interpretation

Think of it like a medical test result breakdown: not just "90% accurate," but "90% of sick people correctly identified (sensitivity), 95% of healthy people correctly identified (specificity), 10% false alarms, 5% missed cases." The matrix forces you to see the asymetry in errors—missing a cancer case is not the same as a false alarm.

What It Is

• True Positive (TP): Correctly predicted positive
• False Negative (FN): Missed positive (Type II error)
• False Positive (FP): False alarm (Type I error)
• True Negative (TN): Correctly predicted negative

WHY this structure? Rows = reality, columns = predictions. Entry $(i, j)$ = "true class $i$, predicted as class $j$." Perfect classifier → diagonal matrix (all off-diagonal = 0).

Deriving Core Metrics from First Principles

Every metric is a ratio of cell counts, designed to answer a specific question.

Accuracy

WHY? "What fraction of all predictions were correct?" Sum diagonal (correct) over total.

WHEN IT FAILS: Imbalanced classes. If 95% of emails are not spam, predicting "not spam" for everything gives 95% accuracy but misses all spam. Accuracy is blind to class importance.

Precision (Positive Predictive Value)

DERIVATION: You predicted positive $(\text{TP} + \text{FP})$ times. How many were actually positive? $\frac{\text{TP}}{\text{TP} + \text{FP}}$.

ANSWERS: "When I say 'positive,' how often am I right?" Measures trustworthiness of positive predictions.

WHY THIS MATTERS: Spam filter—if precision is low, many legitimate emails are marked spam (user frustration). Medical screening—low precision means many healthy people subjected to invasive follow-up tests.

Recall (Sensitivity, True Positive Rate)

DERIVATION: There are $(\text{TP} + \text{FN})$ actual positives. How many did you catch? $\frac{\text{TP}}{\text{TP} + \text{FN}}$.

ANSWERS: "Of all positives, how many did I find?" Measures completeness of detection.

WHY THIS MATTERS: Fraud detection—high recall means few frauds slip through. Cancer screening—high recall means few cases missed (critical for life-threatening conditions).

Specificity (True Negative Rate)

DERIVATION: There are $(\text{TN} + \text{FP})$ actual negatives. How many did you correctly identify? $\frac{\text{TN}}{\text{TN} + \text{FP}}$.

ANSWERS: "Of all negatives, how many did I correctly reject?" Measures ability to avoid false alarms.

COMPLEMENTS RECALL: High recall but low specificity = "crying wolf" (you catch everything but flag tons of false positives).

F1-Score

WHY HARMONIC MEAN? Arithmetic mean $\frac{P + R}{2}$ is too forgiving. If $P = 100\%, R = 10\%$, arithmetic mean = 55% (misleading). Harmonic mean = 18% (penalizes imbalance).

DERIVATION OF SECOND FORM: $F_1 = \frac{2}{\frac{1}{P} + \frac{1}{R}} = \frac{2}{\frac{\text{TP} + \text{FP}}{\text{TP}} + \frac{\text{TP} + \text{FN}}{\text{TP}}} = \frac{2\text{TP}}{\text{TP} + \text{FP} + \text{TP} + \text{FN}} = \frac{2\text{TP}}{2\text{TP} + \text{FP} + \text{FN}}$

WHY THIS STEP? Harmonic mean definition: $\frac{2}{\frac{1}{a} + \frac{1}{b}}$. Substitute precision and recall, simplify denominators, multiply through byTP.

Multi-Class Confusion Matrix

For $C$ classes, the matrix is $C \times C$. Entry $M[i, j]$ = number of times true class $i$ was predicted as class $j$.

Reading this:

• Diagonal (85, 90, 92) = correct predictions
• Off-diagonal = confusion patterns
• Row 1 (cat): 10 cats confused as dogs, 5 as birds
• Column 1: 8 dogs, 3 birds misclassified as cats

PER-CLASS METRICS: For class "Cat" ($i = 1$):

• $TP_{\text{cat}} = 85$ (diagonal)
• $FN_{\text{cat}} = 10 + 5 = 15$ (row 1, off-diagonal)
• $FP_{\text{cat}} = 8 + 3 = 11$ (column 1, off-diagonal)
• $TN_{\text{cat}} = (90 + 2+ 5 + 92) = 189$ (all entries not in row/column 1)

$\text{Precision}_{\text{cat}} = \frac{85}{85 + 11} = \frac{85}{96} = 0.885$ $\text{Recall}_{\text{cat}} = \frac{85}{85 + 15} = \frac{85}{100} = 0.85$

WHY THIS STEP? For class $i$, $TP$ is diagonal entry, $FN$ is row sum minus diagonal, $FP$ is column sum minus diagonal, $TN$ is everything else.

MACRO vs MICRO AVERAGING:

• Macro: Compute metric per class, then average (treats classes equally)
• Micro: Sum all TP, FP, FN across classes, compute once (weights by class size)

Worked Example: Medical Diagnosis

Model predictions:

• 80 sick correctly identified (TP)
• 20 sick missed (FN)
• 50 healthy flaged as sick (FP)
• 850 healthy correctly cleared (TN)

Confusion matrix:

Metrics:

$\text{Accuracy} = \frac{80 + 850}{1000} = 0.93 \text{ (93\%)}$

WHY THIS STEP? Sum diagonal (correct), divide by total. Looks great, but…

$\text{Precision} = \frac{80}{80 + 50} = \frac{80}{130} = 0.615 \text{ (61.5\%)}$

WHY THIS STEP? Of 130 "sick" predictions, only 80 were right. 61.5% trustworthiness—if you test positive, only ~62% chance you're actually sick.

$\text{Recall (Sensitivity)} = \frac{80}{80 + 20} = \frac{80}{100} = 0.80 \text{ (80\%)}$

WHY THIS STEP? Of 100 actual sick, caught 80. Missed 20% of cases—20 sick people sent home!

$\text{Specificity} = \frac{850}{850 + 50} = \frac{850}{900} = 0.944 \text{ (94.4\%)}$

WHY THIS STEP? Of 900 healthy, correctly cleared 850. 5.6% false alarm rate.

TRADEOFF: High specificity (few false alarms), but moderate recall (mises 20% of sick). In medicine, you'd likely lower the threshold to catch more sick (boost recall), accepting more false alarms (lower precision/specificity).

Common Mistakes

Why it feels right: 95% correct sounds great.

The trap: Accuracy is class-blind. With 100 samples, 95 negative, 5 positive—predicting "all negative" = 95% accuracy, but 0% recall (missed every positive).

Example: Fraud detection with 1% fraud rate. "No fraud" everywhere = 99% accuracy, but useless (catches zero fraud).

FIX: Always check precision/recall/F1, especially with imbalanced data. Accuracy alone is meaningful only when classes are balanced.

Why it feels right: Both useTP, both are percentages, easy to mix up.

The trap:

• Precision = "Of my positive predictions, how many were right?" (column view)
• Recall = "Of actual positives, how many did I find?" (row view)

Mnemonic: Precision = **redicted positives (denominator). Recall = Real positives (denominator).

Example: Search engine—high precision = top results are relevant (few junk). High recall = most relevant docs found (few missed). You want both.

FIX: Anchor to the denominator. Precision = $\frac{\text{TP}}{\text{predicted pos}}$, Recall = $\frac{\text{TP}}{\text{actual pos}}$.

Why it feels right: Summary metrics (F1, accuracy) are easier to report.

The trap: You lose where the model fails. Maybe it confuses "cat" with "dog" but never with "bird"—this pattern is invisible a single F1 score.

Example: Multi-class classifier with F1 = 0.85. Confusion matrix reveals: never confuses A↔B, but always confuses C↔D. You'd focus retraining on C/D discrimination, but F1 alone wouldn't tell you.

FIX: Always visualize the matrix for multi-class. Look for systematic confusions (similar classes, data leakage patterns).

The Precision-Recall Tradeoff

WHY? Lower threshold = predict positive more liberally. You'll catch more true positives (↑ recall), but also flag more negatives as positive (↓ precision).

The curve: Plot precision vs recall at all thresholds Precision-Recall curve. Area under curve (AUC-PR) = single-number summary of model quality across thresholds.

Feynman Explanation

• True Positive (TP): You saw someone drowning and saved them. ✓
• False Negative (FN): Someone was drowning, but you missed them. ✗ (worst error—someone died)
• False Positive (FP): You jumped in to "save" someone who was just swimming. ✗ (embarrassing, wastes time)
• True Negative (TN): Person swimming safely, you left them alone. ✓

Accuracy = "What % of the time was I right?" (TP + TN) / total. But if only1 in 100 swimmers is drowning, you could just sit there, do nothing, and be 99% "accurate" while everyone drowns. Useless.

Recall = "Of people who were actually drowning, how many did I save?" If recall is 70%, you missed 30% of drowning people. That's bad.

Precision = "When I jumped in to save someone, how often were they actually drowning?" If precision is 50%, half your rescues were false alarms (you're wasting energy on safe swimmers).

The tradeoff: If you're paranoid and jump in every time someone splashes (low threshold), you'll save everyone (high recall) but annoy lots of safe swimmers (low precision). If you're cautious and only jump for obvious drowning (high threshold), fewer false alarms (high precision) but you miss subtle cases (low recall).

The confusion matrix shows you exactly how many of each type of mistake you made. Then you decide: in lifeguarding, missing a drowning person is worse than a false alarm, so you'd optimize for high recall (lower threshold), even if precision drops.

Mnemonic

Confusion matrix corners (clockwise from top-left): "TP, FP, TN, FN" → "True, False, True, False" + "Positive, Positive, Negative, Negative"

Or: "Two Pints, Two Notes" (TP, FP, TN, FN) if you like beer and music.

Connections

• Precision and Recall – detailed derivations, F-beta score
• ROC Curve and AUC – alternative evaluation for threshold-based models
• Class Imbalance Handling – why accuracy fails, how to rebalance
• Multi-class Classification Metrics – macro/micro averaging, per-class analysis
• Cost-Sensitive Learning – assigning different costs to FP vs FN
• Model Selection Criteria – how to choose between models with different confusion matrices
• Binary Classification Thresholds – moving the threshold to optimize for business goals

#flashcards/ai-ml

What are the four cells of a binary confusion matrix? :: True Positive (TP), False Negative (FN), False Positive (FP), True Negative (TN)

Concept Map

Hinglish (regional understanding)