Exercises — Accuracy, precision, recall, F1-score

WHAT we want: fraction of all predictions that were correct. WHY accuracy: the question says "how many did it get right overall" — no mention of a class we care about more, so we treat every cell equally. Correct cells are the two "true" ones: $TP+TN=60+110=170$. Total = all four cells $=60+110+10+20=200$. $\text{Acc}=\frac{170}{200}=0.85$ Answer: 0.85 (85%).

Precision asks: of everything I called positive, how much was right? The "called positive" column is $TP+FP=60+10=70$. $P=\frac{60}{70}=0.857$ Recall asks: of everything that actually was positive, how much did I catch? Actual positives are $TP+FN=60+20=80$. $R=\frac{60}{80}=0.75$ Answers: $P=0.857$, $R=0.75$. Notice the denominators differ — that's the whole point of these being two different questions.

The trick: the first word (True/False) tells you if the model was right; the second word (Positive/Negative) is what the model said.

• (a) said positive, was right → TP
• (b) said positive, was wrong → FP (Type I error, a false alarm)
• (c) said negative, was wrong → FN (Type II error, a miss)
• (d) said negative, was right → TN

$\text{Acc}=\frac{30+940}{1000}=\frac{970}{1000}=0.97$ $P=\frac{30}{30+20}=\frac{30}{50}=0.60$ $R=\frac{30}{30+10}=\frac{30}{40}=0.75$ F1 — the harmonic mean, which drags toward the smaller of P and R: $F1=\frac{2\cdot 0.60\cdot 0.75}{0.60+0.75}=\frac{0.90}{1.35}=0.667$ Answers: Acc $=0.97$, $P=0.60$, $R=0.75$, $F1=0.667$. Note how accuracy (0.97) looks glorious while F1 (0.667) is honest — because the negatives dominate.

WHY this form exists: it skips computing P and R separately — you plug the raw counts straight in. It's algebraically identical (see the parent's derivation). $F1=\frac{2\cdot 30}{2\cdot 30+20+10}=\frac{60}{60+30}=\frac{60}{90}=0.667$ Match confirmed: $0.667$. Same answer, fewer steps — handy when you already have the counts.

$P=\frac{1}{1+0}=1.0(\text{every alarm it raised was correct})$ $R=\frac{1}{1+99}=0.01(\text{it caught 1 of 100 real positives})$ $F1=\frac{2\cdot 1.0\cdot 0.01}{1.0+0.01}=\frac{0.02}{1.01}=0.0198$ Answers: $P=1.0$, $R=0.01$, $F1\approx 0.0198$. This is the classic case: perfect precision, catastrophic recall. The arithmetic mean would give $0.505$ (misleading); the harmonic mean gives $\approx 0.02$ (honest) because it refuses to reward one metric when the other is near zero.

Model A: $P_A=\frac{18}{18+30}=\frac{18}{48}=0.375,R_A=\frac{18}{18+2}=\frac{18}{20}=0.90$ $F{1}_A=\frac{2\cdot 0.375\cdot 0.90}{0.375+0.90}=\frac{0.675}{1.275}=0.529$ Model B: $P_B=\frac{14}{14+3}=\frac{14}{17}=0.824,R_B=\frac{14}{14+6}=\frac{14}{20}=0.70$ $F{1}_B=\frac{2\cdot 0.824\cdot 0.70}{0.824+0.70}=\frac{1.153}{1.524}=0.757$ By F1: Model B wins ($0.757>0.529$). For cancer screening specifically: a missed cancer (FN) is catastrophic, so we prioritise recall. Model A catches 18/20 sick patients (recall 0.90); Model B misses 6 (recall 0.70). Model A is safer for screening despite its worse F1 — because F1 balances both errors equally, but here the two errors are not equally costly. Lesson: the "better" model depends on which error hurts more, not on a single blended number.

$P=\frac{TP}{TP+FP},R=\frac{TP}{TP+FN}$ If $FP=FN$, the two denominators are identical and the numerators are both $TP$, so $P=R$. Call this common value $x$. Then $F1=\frac{2x\cdot x}{x+x}=\frac{2x^2}{2x}=x=P=R.$ Meaning: when your two error types are equal in count, all three of P, R, F1 collapse to one number — there's nothing for F1 to "balance". Model B in Example 4 of the parent ($FP=FN=20$) is exactly this case.

Key insight: F1 depends on P and R only — the raw counts have already been folded into those two numbers. So yes: $F1=\frac{2PR}{P+R}=\frac{2\cdot 0.8\cdot 0.5}{0.8+0.5}=\frac{0.80}{1.30}=0.615$ Answer: $F1\approx 0.615$. You cannot recover accuracy though — that needs TN, which P and R never see. This is why F1 survives on imbalanced data: it ignores the (often huge) TN count entirely.

Step 1 — translate the requirement into recall. "At most 1 miss out of 20" means $FN\le 1$, so $TP\ge 19$. At the boundary $TP=19$, $FN=1$: $R=\frac{19}{19+1}=\frac{19}{20}=\mathrm{0.95.}$ So the recall floor is 0.95. Step 2 — precision at $FP=10$: $P=\frac{19}{19+10}=\frac{19}{29}=\mathrm{0.655.}$ Step 3 — F1: $F1=\frac{2\cdot 0.655\cdot 0.95}{0.655+0.95}=\frac{1.245}{1.605}=\mathrm{0.776.}$ Answers: recall floor $=0.95$, $P\approx 0.655$, $F1\approx 0.776$. This is exactly how Threshold Optimization works: fix the metric a stakeholder cares about (here recall $\ge 0.95$), then move the decision threshold to maximise the other.

(a) Accuracy: $\text{Acc}=\frac{0+9990}{10000}=0.999=99.9\%.$ (b) Recall: $R=\frac{0}{0+10}=0.$ Precision is $\frac{0}{0+0}=\frac{0}{0}$ — undefined. By the standard scikit-learn convention we define precision $=0$ when there are no positive predictions (the model raised zero correct alarms). Then $F1=\frac{2\cdot 0\cdot 0}{0+0}\to \text{define as }0.$ So recall $=0$, F1 $=0$. (c) A model can score 99.9% accuracy while being completely useless on the class that matters — proof that on imbalanced data accuracy is the wrong headline metric; recall/F1 expose the failure.

(a) $\beta =1$: $F_1=(1+1)\frac{PR}{1\cdot P+R}=\frac{2PR}{P+R},$ which is exactly the F1 formula. ✓ (b) With $P=0.6$, $R=0.9$: $F_2=(1+4)\frac{0.6\cdot 0.9}{4\cdot 0.6+0.9}=5\cdot \frac{0.54}{3.3}=\frac{2.70}{3.3}=\mathrm{0.818.}$ Ordinary F1 for the same numbers: $F1=\frac{2\cdot 0.6\cdot 0.9}{0.6+0.9}=\frac{1.08}{1.5}=\mathrm{0.72.}$ $F_2=0.818>F1=0.72$. Why: here recall (0.9) is higher than precision (0.6). $F_2$ leans toward recall, so it rewards this model more than the balanced F1 does. If instead precision had been the higher number, $F_2$ would have scored lower than F1 — because it would be down-weighting the metric that happened to be strong. The moral: $\beta$ lets you tune which mistake the score forgives.

Accuracy uses all four cells including the huge TN — that's its weakness on imbalance ::: True; precision and recall both ignore TN, which is why F1 survives imbalanced data The harmonic mean is used for F1 because it ::: collapses toward the smaller of P and R, punishing any model that is lopsided To compute F1 you must have the raw counts TP, FP, FN ::: False — $F1=\frac{2PR}{P+R}$ needs only precision and recall