2.6.6 · D5Model Evaluation & Selection
Question bank — Confusion matrix interpretation
Before we start, one shared picture to anchor every symbol used below.
Figure description: a 2×2 grid. Rows are labelled Actual + (top) and Actual − (bottom); columns are labelled Predicted + (left) and Predicted − (right). Top-left = TP (green, true +, predicted +), top-right = FN (red, true +, predicted − , a miss), bottom-left = FP (yellow, true −, predicted +, a false alarm), bottom-right = TN (blue, true −, predicted −). Four arrows anchor all four ratios: a yellow arrow shows precision read DOWN the left (predicted-positive) column; a green arrow shows recall read ACROSS the top (actual-positive) row; a blue arrow shows specificity read ACROSS the bottom (actual-negative) row; a red arrow shows NPV read UP the right (predicted-negative) column.
Every metric this page tests is just a ratio of these four cells. Here they all are, defined once, so nothing below is used before it is stated:
The same cells also give the two error rates that the ROC curve is built from — the complements of recall and specificity:
Two more, built from the cells:
Figure description: two curves plotted for a fixed precision as recall varies from 0 to 1. The straight yellow line is the arithmetic mean ; the curved green line dipping below it is the harmonic mean . The gap shows how the harmonic mean is always pulled toward whichever of , is smaller.
Figure description: as the decision threshold slides from high to low, recall (green) climbs toward 1 while precision (yellow) falls — a visual of the precision–recall trade-off that lowering a threshold produces.
Recall What do the four cells mean? (reveal to warm up)
Row = the truth, column = the model's guess ::: TP = truly positive, guessed positive. FN = truly positive, guessed negative (a miss). FP = truly negative, guessed positive (a false alarm). TN = truly negative, guessed negative.
True or false — justify
A model with 99% accuracy is always better than one with 90% accuracy.
False — accuracy is class-blind. On a 1% fraud dataset, "predict never-fraud" scores 99% while catching zero frauds; the 90% model might actually find them.
Precision and recall both have TP in the numerator, so they always move together.
False — they share a numerator but have different denominators (predicted-positives vs actual-positives). Lowering the threshold usually raises recall while lowering precision.
A perfect confusion matrix is a diagonal matrix.
True — every off-diagonal cell counts a mistake (true class guessed as class ), so zero errors means only the diagonal is non-zero.
If accuracy is 100%, then precision, recall, and specificity are all 100% too.
True — 100% accuracy forces FP = FN = 0, and every metric that divides TP or TN by a sum containing those zeros (see the formula box above) collapses to 1.
The F1-score can be higher than both precision and recall.
False — the harmonic mean of two numbers never exceeds the larger, and always sits at or below the smaller of the two, so F1 can never beat both.
FPR and FNR always add up to 1.
False — FPR = FP/(FP+TN) lives in the negative row, FNR = FN/(TP+FN) lives in the positive row; they come from different denominators and are independent. It is FPR + specificity = 1 and FNR + recall = 1 that hold.
Swapping which class you call "positive" leaves precision unchanged.
False — precision and recall are defined relative to the positive class; relabelling swaps the roles of the cells and generally changes both. Accuracy, however, stays the same.
Specificity and recall answer the same question for different classes.
True in spirit — recall is "TP-rate for positives," specificity is "TN-rate for negatives." Specificity is just recall computed with the negative class treated as positive.
Precision and NPV are the same metric with the labels swapped.
True in structure — precision is "of predicted-positives, how many right?" and NPV is "of predicted-negatives, how many right?"; each trusts one predicted column. Their values usually differ because the two columns hold different cells.
A model can have 100% recall and still be nearly useless.
True — predicting "positive" for everyone catches every real positive (recall = 100%) but has terrible precision ( with a huge FP) and FPR = 100%, flooding you with false alarms.
Adding more TN to a confusion matrix always raises precision.
False — precision is and contains no TN term at all. Piling up true negatives changes accuracy, specificity, FPR, and NPV but never touches precision.
Spot the error
"Precision equals TP divided by all actual positives."
Wrong denominator — that formula is recall. Precision divides by all predicted positives, .
"We got 93% accuracy on the disease screen, so the model is safe to deploy."
The trap: 93% accuracy hid a 20% miss rate (recall = 80%, so FNR = 20%) — 20 sick patients sent home. Accuracy alone hides the asymmetric cost of an FN.
"To improve recall we should reduce false positives."
Recall depends on FN, not FP: . To raise recall you must reduce misses (FN), typically by lowering the decision threshold. Reducing FP raises precision instead.
"FPR is FP divided by all predicted positives."
Wrong denominator — FPR = FP/(FP+TN) divides by all actual negatives, not predicted positives. Dividing FP by predicted positives () gives , a different quantity.
"For the cat class, FP is the sum of row 1 off the diagonal."
Reversed — that sum is FN (real cats guessed as something else). FP for a class is the column off-diagonal (other animals guessed as cat).
"Macro-averaging weights each class by how many samples it has."
That describes micro-averaging. Macro computes the metric per class and averages equally, giving small rare classes the same vote as big ones.
"F1 = (Precision + Recall) / 2."
That is the arithmetic mean, which is too forgiving. F1 is the harmonic mean, , which crashes toward the smaller value when the two are unbalanced.
"A 60% precision means 60% of sick people were found."
Confuses precision with recall. 60% precision means 60% of the people flagged sick really were sick; how many sick people were found is recall.
"Specificity is TN divided by all predicted negatives."
That is NPV, not specificity. Specificity divides TN by all actual negatives (); NPV divides TN by all predicted negatives ().
Why questions
Why does accuracy fail on imbalanced data but precision/recall survive?
Accuracy pools all cells, so the huge TN count of the majority class drowns out errors on the rare class. Precision and recall ignore TN entirely, so they stay sensitive to the minority.
Why use the harmonic mean for F1 instead of the arithmetic mean?
The harmonic mean punishes imbalance: if and , arithmetic gives a rosy 55% but harmonic gives about 18%, refusing to reward a model that is great on one axis and useless on the other.
Why does the ROC curve plot FPR on the x-axis rather than precision?
FPR and TPR both ignore how many samples are positive vs negative (each is a within-class rate), so the ROC curve stays stable under class imbalance; precision, which mixes classes in its denominator, would shift as the base rate changes.
Why is a false negative often "worse" than a false positive in medicine?
An FN is a missed sick patient who goes untreated; an FP is a healthy patient sent for extra tests. The costs are asymmetric, which is why cost-sensitive thinking beats raw accuracy.
Why does lowering the classification threshold trade precision for recall?
A lower threshold labels more cases positive, so you catch more true positives (recall ↑, FNR ↓) but also sweep in more false alarms (precision ↓, FPR ↑). This trade is exactly what the ROC curve traces out.
Why do rows mean "truth" and columns mean "prediction," not the reverse?
It is a fixed convention so that each entry reads "true class was predicted as class ." Fixing it lets FN always be a row-wise miss and FP a column-wise false alarm — flipping it silently swaps every metric.
Why is TN irrelevant to both precision and recall?
Both metrics live entirely in the positive-prediction world: precision asks about predicted-positives, recall about actual-positives. True negatives never enter either denominator, so they cannot influence them.
Why can two models have identical accuracy but wildly different F1?
Accuracy fixes only the total correct (TP + TN); it says nothing about how the errors split between FP and FN. One model may miss many positives (low recall) while another floods false alarms (low precision) — same accuracy, different F1.
Why does NPV matter when precision looks fine?
Precision only vouches for positive predictions; NPV vouches for negative ones. In screening, a high NPV means "if we cleared you, you're probably really healthy" — the reassurance a patient sent home actually needs.
Edge cases
What is precision when the model predicts "negative" for every sample?
TP = FP = 0, so precision is — undefined. By convention it is reported as 0 (or NaN), signalling the model never risked a positive claim.
What is recall (and therefore FNR) when there are zero actual positives in the test set?
, so recall is — undefined, and FNR = 1 − recall is undefined too, because "how many positives did you find/miss" is meaningless when none exist.
What is specificity (and FPR) when there are zero actual negatives ()?
Specificity is — undefined, and FPR = 1 − specificity is likewise undefined. With no real negatives there is nothing to correctly reject or falsely flag; report both as undefined, not 0.
What happens to F1 when either precision or recall is exactly zero?
F1 collapses to 0. The harmonic mean has a product in the numerator, so any zero factor zeroes the whole score, no matter how high the other value.
For a perfectly balanced 50/50 dataset, is accuracy still misleading?
Much less so — with balanced classes, accuracy weights both errors fairly and roughly tracks the average of recall and specificity. The class-blindness trap only bites when one class dominates.
If FP = FN, does that mean precision equals recall?
Yes — both are , so equal off-diagonal error counts force . This is exactly the balance point where F1 equals both of them.
What is NPV when the model predicts "positive" for every sample?
TN = FN = 0, so NPV is — undefined, mirroring the precision edge case: the model never made a negative prediction to be right or wrong about.
In a multi-class matrix, what is TN for a single class?
Everything not in that class's row or column — all cells where neither the truth nor the prediction was that class. For per-class scoring this is the "everyone else got it right elsewhere" bucket.
What does a confusion matrix look like for a model that always predicts one fixed class?
A single non-empty column (the predicted class) with all mass stacked there; every other prediction column is zero. Recall is 100% for that class and 0% for all others — a red flag for model selection.