Accuracy, precision, recall, F1-score

The Foundation: Confusion Matrix

• True Positive (TP): Model predicted positive, actually positive
• True Negative (TN): Model predicted negative, actually negative
• False Positive (FP): Model predicted positive, actually negative (Type I error)
• False Negative (FN): Model predicted negative, actually positive (Type II error)

Why this matters: Every classification metric is computed from these four numbers. Understanding the confusion matrix means understanding what each metric emphasizes.

Metric 1: Accuracy

From first principles: We want $\frac{\text{correct predictions}}{\text{total predictions}}$.

Correct predictions = TP + TN (got positives right + got negatives right)
Total predictions = TP + TN + FP + FN (all four quadrants)

$\text{Accuracy} = \frac{\text{TP} + \text{TN}}{\text{TP} + \text{TN} + \text{FP} + \text{FN}}$

Interpretation: Accuracy = 0.95 means 95% of predictions are correct.

$\text{Accuracy} = \frac{45 + 48}{45 + 48 + 2 + 5} = \frac{93}{100} = 0.93$

Why 93%? Because 93 out of 100 predictions matched reality.

Why it feels right: 95% sounds impressive — only 5% errors.

The problem: Consider fraud detection with 1000 transactions: 990 legitimate, 10 fraudulent.
A lazy model that predicts "legitimate" for everything:

• TP = 0 (catches no fraud)
• TN = 990 (correctly identifies all legitimate)
• FP = 0
• FN = 10 (mises all fraud)

$\text{Accuracy} = \frac{0 + 990}{1000} = 0.99 = 99\%$

The fix: On imbalanced datasets, accuracy is misleading because the majority class dominates. A useless model can have high accuracy. Use precision/recall instead.

Metric 2: Precision

From first principles: We want $\frac{\text{correct positive predictions}}{\text{all positive predictions}}$.

Correct positive predictions = TP
All positive predictions = TP + FP (actual positives + false alarms)

$\text{Precision} = \frac{\text{TP}}{\text{TP} + \text{FP}}$

Interpretation: Precision = 0.8 means when the model says "positive", it's right 80% of the time.

Alternative name: Positive Predictive Value (PPV)

$\text{Precision} = \frac{80}{80 + 20} = \frac{80}{100} = 0.8$

Why this matters: 20% of flaged emails are false alarms. Users will miss important emails! For spam filters, high precision is critical — better to let spam through than block important mail.

Why these specific numbers?

• TP = 80: The model correctly identified 80 spam emails
• FP = 20: The model wrongly flagged 20 legitimate emails as spam
• Total predicted positive = 100 (what the model claimed was spam)
• Precision tells us: "Of the 100 emails model called spam, only 80 actually were spam"

Metric 3: Recall

From first principles: We want $\frac{\text{positives we found}}{\text{all actual positives}}$.

Positives we found = TP
All actual positives = TP + FN (found positives + missed positives)

$\text{Recall} = \frac{\text{TP}}{\text{TP} + \text{FN}}$

Interpretation: Recall = 0.9 means the model catches 90% of actual positives.

Alternative names: Sensitivity, True Positive Rate (TPR)

$\text{Recall} = \frac{18}{18 + 2} = \frac{18}{20} = 0.9$

Why 90%? The model found 18 out of 20 actual cancer cases. It missed 2 (10%).

Why recall matters here: Missing cancer (false negative) is catastrophic. For medical screening, high recall is critical — better to have false alarms than miss disease.

Why these specific numbers?

• TP = 18: Correctly identified 18 of the sick patients
• FN = 2: Missed 2 sick patients (they tested negative but had cancer)
• Total actual positive = 20 (ground truth)
• Recall tells us: "Of the 20 patients who actually had cancer, the model detected 18"

Why it feels right: Both involveTP in the numerator and sound similar.

The problem: They have different denominators and measure different failure modes:

• Precision: "Of what I claimed was positive, how much was right?" (denominator = predicted positive)
• Recall: "Of what actually was positive, how much did I find?" (denominator = actual positive)

Example to separate them:

• Model predicts positive only once, and it's correct: TP=1, FP=0, FN=99
  • Precision = 1/1 = 100% (perfect! every prediction was right)
  • Recall = 1/100 = 1% (terrible! missed 99%)

The fix: Precision asks "How reliable are my alarms?", recall asks "How many real cases did I catch?"

Metric 4: F1-Score

From first principles: We want the harmonic mean of precision and recall, not the arithmetic mean. Why harmonic?

If we used arithmetic mean: $\frac{P + R}{2}$
Problem: $P=1, R=0.01 \rightarrow \text{mean} = 0.505$ (looks decent, but recall is terrible!)

Harmonic mean: $\frac{2}{\frac{1}{P} + \frac{1}{R}}$ (reciprocals average, then flip back)

This penalizes extreme imbalance — if either P or R is low, F1 is low.

Simplifying: $\text{F1} = \frac{2}{\frac{1}{P} + \frac{1}{R}} = \frac{2PR}{P+R} = \frac{2 \cdot \text{TP}}{2 \cdot \text{TP} + \text{FP} + \text{FN}}$

Why the last form? Substitute $P = \frac{\text{TP}}{\text{TP}+\text{FP}}$ and $R = \frac{\text{TP}}{\text{TP}+\text{FN}}$:

$\frac{2PR}{P+R} = \frac{2 \cdot \frac{\text{TP}}{\text{TP}+\text{FP}} \cdot \frac{\text{TP}}{\text{TP}+\text{FN}}}{\frac{\text{TP}}{\text{TP}+\text{FP}} + \frac{\text{TP}}{\text{TP}+\text{FN}}}$

Multiply numerator and denominator by $(\text{TP}+\text{FP})(\text{TP}+\text{FN})$:

$= \frac{2 \cdot \text{TP}^2}{(\text{TP}+\text{FN})\text{TP} + (\text{TP}+\text{FP})\text{TP}} = \frac{2 \cdot \text{TP}^2}{\text{TP}(2\text{TP} + \text{FP} + \text{FN})} = \frac{2\text{TP}}{2\text{TP}+\text{FP}+\text{FN}}$

Model B (balanced):

• TP=70, FP=20, FN=20
• Precision = 70/90 = 0.778
• Recall = 70/90 = 0.778
• F1 = 2(0.778)(0.778)/(0.778+0.778) = 0.778

Why Model B has higher F1: Despite lower precision than Model A, Model B's balance gives it a better F1. The harmonic mean rewards balance.

Step-by-step for Model A F1:

1. Compute precision: $\frac{50}{50+5} = \frac{50}{55} = 0.909$
2. Compute recall: $\frac{50}{50+40} = \frac{50}{90} = 0.556$
3. Apply harmonic mean: $\frac{2 \times 0.909 \times 0.556}{0.909 + 0.556} = \frac{1.011}{1.465} = 0.690$

The Precision-Recall Tradeoff

Lower threshold → predicts positive more often:

• Catches more true positives → recall ↑
• But also more false alarms → precision ↓

Higher threshold → predicts positive less often:

• Fewer false alarms → precision ↑
• But mises more true positives → recall ↓

You cannot maximize both simultaneously (except trivially at 100% perfect classification). This is why F1-score exists — to find a reasonable balance.

Dataset: 10 samples (5 actually positive, 5 actually negative)
Scores: [0.9, 0.8, 0.6, 0.4, 0.3] (positives), [0.7, 0.5, 0.4, 0.2, 0.1] (negatives)

Threshold = 0.5:

• Predicted positive: 0.9, 0.8, 0.7, 0.6, 0.5
• TP=4, FP=2, FN=1
• Precision = 4/6 = 0.667
• Recall = 4/5 = 0.8

Threshold = 0.75:

• Predicted positive: 0.9, 0.8
• TP=2, FP=0, FN=3
• Precision = 2/2 = 1.0
• Recall = 2/5 = 0.4

Why? Higher threshold → fewer predictions → fewer false positives (precision up) but more missed positives (recall down).

When to Use Which Metric

The F-beta score generalizes F1: $F_\beta = (1+\beta^2) \frac{PR}{\beta^2 P + R}$

• $\beta < 1$: favors precision
• $\beta > 1$: favors recall
• $\beta = 1$: F1-score (equal weight)

Multi-Class Extension

For multi-class classification (>2 classes), compute metrics per class then aggregate:

1. Macro-average: Compute metric for each class, then average (treats all classes equally)
2. Micro-average: Pool allTP/FP/FN across classes, then compute metric (treats all samples equally)
3. Weighted-average: Weight each class by its frequency

Example intuition: 3 classes (A: 100samples, B: 50 samples, C: 10 samples)

• Macro: (metric_A + metric_B + metric_C)/3 (class C counts as much as A)
• Micro: aggregate all TP/FP/FN first (class A dominates due to size)
• Weighted: weight by100, 50, 10 (balanced approach)

Accuracy is like "How many questions did I grade correctly out of all questions?" If95 questions have answer FALSE and only 5 have answer TRUE, and I just mark everything FALSE, I get 95% accuracy — but I learned nothing about the TRUE questions!

Precision is "When I circle a paper and say 'This kid was guessing!', how often am I right?" If I accuse 10 kids and 8 were actually guessing, that's 80% precision. The 2 I wrongly accused are false alarms.

Recall is "Of all the kids who were actually guessing, how many did I catch?" If 20 kids were guessing and I only caught 15, that's 75% recall. I missed 5.

F1 is like a report card that combines both: "Am I good at catching guessers AND not accusing innocent kids?" If I'm great at one but terrible at the other, my F1 is low. I need to be balanced.

The tradeoff: If I accuse EVERYONE of guessing (to catch all guessers), I have perfect recall but terrible precision (too many false accusations). If I only accuse people when I'm 100% sure, I have perfect precision but terrible recall (I miss most guessers). F1 helps find the sweet spot.

Denominator trick:

• Precision: denominator has P (predicted positive = TP + FP)
• Recall: denominator has N (actual positive = TP + FN, where FN = False Negative)

Harm mnemonic:

• Precision: minimize harm from false alarms (FP in denominator)
• Recall: minimize harm from mises (FN in denominator)

Connections

• Confusion Matrix - the foundation for all classification metrics
• ROC Curve and AUC - visualizes precision-recall tradeoff across all thresholds
• Cross-Validation - use these metrics to compare models fairly
• Class Imbalance Handling - why accuracy fails and sampling/weighting help
• Cost-Sensitive Learning - when false positives and false negatives have different costs
• Multi-Label Classification - extending these metrics to multiple labels per sample
• Threshold Optimization - finding the best decision threshold for your use case
• Precision-Recall Curve - alternative to ROC for imbalanced datasets

#flashcards/ai-ml

What are the four outcomes in a confusion matrix? :: True Positive (predicted +, actually +), True Negative (predicted -, actually -), False Positive (predicted +, actually -), False Negative (predicted -, actually +)

Derive the accuracy formula from first principles :: Accuracy = (correct predictions)/(total predictions) = (TP + TN)/(TP + TN + FP + FN), where correct = got positives right + got negatives right

Derive the F1-score formula :: F1 = 2/(1/P + 1/R) = 2PR/(P+R) = 2TP/(2TP + FP + FN). It's the harmonic mean of precision and recall.

Concept Map

Hinglish (regional understanding)