Confusion matrix interpretation: Exercises — AI-ML deep dive

This is the practice ground for Confusion matrix interpretation. Every problem states its numbers cleanly, then hides a full worked solution inside a collapsible callout so you can self-test. Levels rise from "can you name the cell?" (L1) to "can you design a metric for a business goal?" (L5).

Figure — Confusion matrix interpretation

Figure 1 (described): a 2×2 grid. Rows are reality (Actual + on top, Actual − below), columns are predictions (Predicted + left, Predicted − right). The top-left cell is TP drawn in red — the "key hit" — while FN (top-right), FP (bottom-left) and TN (bottom-right) are in black. If the image fails to load: TP = actual+/pred+, FN = actual+/pred−, FP = actual−/pred+, TN = actual−/pred−.

Reminder of the four formulas you will reach for (all are ratios of the counts above):

Common mistake

Watch the zero denominators (edge cases). Every metric here is a ratio, and a ratio blows up when its bottom is $0$ :

Precision is undefined when $T P + F P = 0$ — you never predicted positive at all, so "how trustworthy are my positive calls?" has no answer. By convention libraries return $0$ (or NaN); interpret it as "no positive predictions made," not as "great precision."
Recall is undefined when $T P + F N = 0$ — there were no actual positives in the data, so "how many positives did I catch?" is meaningless. Report it as "N/A — no positives present," never as $1.0$ .
Specificity / FPR are undefined when $T N + F P = 0$ (no actual negatives).
$F_{1}$ is undefined when both precision and recall are $0$ (i.e. $T P = 0$ ); it is then taken as $0$ . Rule of thumb: before dividing, check the denominator; a $0$ there means "the question doesn't apply to this data," not a real score.

Level 1 — Recognition

Exercise 1.1

A model is evaluated. For one test sample the true label is Positive and the model predicted Negative. Name this cell of the confusion matrix and its classical error name.

Recall Solution

WHAT: True = Positive, Predicted = Negative → we missed a real positive. CELL: False Negative (FN). CLASSICAL NAME: Type II error (a "miss"). PICTURE: In the confusion-matrix grid above (Figure 1), rows are reality and columns are predictions; "Actual Pos, Predicted Neg" is the top-right cell.

Exercise 1.2

Given the confusion matrix below, read off TP, FN, FP, TN directly.

Actual Pos Actual Neg Pred Pos 405 Pred Neg 1045

Recall Solution

WHAT each number means (read row = reality, column = prediction):

$T P = 40$ — actual pos, predicted pos.
$F N = 10$ — actual pos, predicted neg (row-1 off-diagonal).
$F P = 5$ — actual neg, predicted pos (row-2, left column).
$T N = 45$ — actual neg, predicted neg. CHECK: Total $= 40 + 10 + 5 + 45 = 100$ samples. Diagonal $(40, 45)$ are the correct ones.

Level 2 — Application

Exercise 2.1

Using the matrix from Exercise 1.2 ( $T P = 40, F N = 10, F P = 5, T N = 45$ ), compute Accuracy, Precision, Recall, and Specificity.

Recall Solution

First, the denominators are all non-zero ( $T P + F P = 45$ , $T P + F N = 50$ , $T N + F P = 50$ ), so every metric is well-defined here. Accuracy — fraction of all predictions correct: $Accuracy = \frac{T P + T N}{100} = \frac{40 + 45}{100} = \frac{85}{100} = 0.85 (85%)$ Precision — of everything I called positive, how many were right? Denominator is predicted positives $= T P + F P = 45$ : $Precision = \frac{40}{40 + 5} = \frac{40}{45} = 0.8 \overline{8} \approx 0.889$ Recall — of every actual positive, how many did I catch? Denominator is actual positives $= T P + F N = 50$ : $Recall = \frac{40}{40 + 10} = \frac{40}{50} = 0.80$ Specificity — of every actual negative, how many did I correctly reject? Denominator $= T N + F P = 50$ : $Specificity = \frac{45}{45 + 5} = \frac{45}{50} = 0.90$

Exercise 2.2

For the same matrix, compute the $F_{1}$ -score two ways (from precision & recall, and from raw counts) and derive why the raw-count shortcut equals the harmonic mean.

Recall Solution

Way 1 — harmonic mean of $P$ and $R$ . Keep $P = \frac{40}{45}$ and $R = \frac{4}{5}$ as exact fractions to avoid rounding drift, then plug into the harmonic mean: $F_{1} = 2 \cdot \frac{P \cdot R}{P + R} = 2 \cdot \frac{\frac{40}{45} \cdot \frac{4}{5}}{\frac{40}{45} + \frac{4}{5}} = 2 \cdot \frac{\frac{160}{225}}{\frac{76}{45}} = \frac{80}{95}$ (If instead you round early — $P \approx 0.889$ , $R = 0.80$ — you get $2 \cdot \frac{0.889 \times 0.80}{0.889 + 0.80} \approx 2 \cdot \frac{0.7111}{1.689} \approx 0.8421$ . Same value, but note these decimals are approximations; the exact fraction $\frac{80}{95}$ is the honest answer.) Way 2 — raw-count shortcut $F_{1} = \frac{2 T P}{2 T P + F P + F N}$ : $F_{1} = \frac{2 \times 40}{2 \times 40 + 5 + 10} = \frac{80}{95} = 0.8421 \dots$ WHY they match (derived here, step by step). Start from the harmonic-mean definition of $F_{1}$ and substitute $P = \frac{T P}{T P + F P}$ , $R = \frac{T P}{T P + F N}$ : $F_{1} = \frac{2}{\frac{1}{P} + \frac{1}{R}} = \frac{2}{\frac{T P + F P}{T P} + \frac{T P + F N}{T P}}$ WHAT we did: flipped each fraction ( $\frac{1}{P} = \frac{T P + F P}{T P}$ , likewise for $R$ ). WHY the harmonic form: the arithmetic mean $\frac{P + R}{2}$ is too forgiving — if $P = 1.0$ but $R = 0.1$ it reports $0.55$ ; the harmonic mean drags the score toward the smaller value, so both must be high. Now add the two fractions over the common denominator $T P$ : $F_{1} = \frac{2}{\frac{( T P + F P ) + ( T P + F N )}{T P}} = \frac{2 T P}{2 T P + F P + F N}$ WHAT we did: combined the numerators, then multiplied top and bottom by $T P$ to clear the nested fraction. That last expression is exactly the shortcut — so the two forms are algebraically identical, not a coincidence. Both give the exact value $\frac{80}{95} \approx 0.842$ .

Level 3 — Analysis

Exercise 3.1

A spam filter processes 10,000 emails. Only 200 are truly spam. The model flags 180 spam correctly, misses 20, and wrongly flags 300 legitimate emails as spam. Build the confusion matrix, then explain why its 96.8% accuracy is misleading.

Recall Solution

Fill the counts. Actual spam $= 200$ , actual legit $= 9800$ .

$T P = 180$ (spam caught), $F N = 20$ (spam missed).
$F P = 300$ (legit flagged), $T N = 9800 - 300 = 9500$ (legit passed).

Actual Spam Actual Legit Pred Spam 180300 Pred Legit 209500

Accuracy: $\frac{180 + 9500}{10000} = \frac{9680}{10000} = 0.968 (96.8%)$ . Precision: $\frac{180}{180 + 300} = \frac{180}{480} = 0.375$ . Recall: $\frac{180}{180 + 20} = \frac{180}{200} = 0.90$ . WHY accuracy misleads: 98% of emails are legit, so a model that just passes almost everything scores high on accuracy automatically — the huge $T N$ dominates. But precision is only 37.5%: nearly two-thirds of things sent to the spam folder are actually legitimate. Accuracy is class-blind; the imbalance hides the real cost. See Class Imbalance Handling.

Exercise 3.2

Same spam data. The team retunes the threshold (see Binary Classification Thresholds) so it now catches all 200 spam ( $F N = 0$ ) but flags 1200 legit emails ( $F P = 1200$ ). Did the model get better? Compare recall, precision, and specificity before vs after.

Recall Solution

New counts: $T P = 200, F N = 0, F P = 1200, T N = 9800 - 1200 = 8600$ .

Metric	Before	After
Recall $\frac{T P}{T P + F N}$	$\frac{180}{200} = 0.90$	$\frac{200}{200} = 1.00$
Precision $\frac{T P}{T P + F P}$	$\frac{180}{480} = 0.375$	$\frac{200}{1400} \approx 0.143$
Specificity $\frac{T N}{T N + F P}$	$\frac{9500}{9800} \approx 0.969$	$\frac{8600}{9800} \approx 0.878$

Interpretation: Recall rose to a perfect $1.0$ — every spam caught. But precision collapsed to $14.3%$ , and specificity dropped. This is the classic precision–recall trade-off: pushing the threshold to catch more positives sweeps in more false alarms. "Better" depends on cost — for a spam filter, deleting real email (low precision) is worse than letting some spam through, so the "after" model is arguably worse. See Precision and Recall.

Level 4 — Synthesis

Exercise 4.1

A 3-class classifier (cat, dog, bird) produces:

True Cat True Dog True Bird Pred Cat 30105 Pred Dog 15405 Pred Bird 5040

Compute per-class precision and recall for Dog, then compute the macro-averaged recall across all three classes. (Total samples $= 150$ .)

Recall Solution

Dog metrics — treat "Dog" as the positive class:

$T P_{dog} = 40$ (diagonal).
$F N_{dog} =$ row-2 off-diagonal $= 10 + 0 = 10$ .
$F P_{dog} =$ column-2 off-diagonal $= 15 + 5 = 20$ . $Precision_{dog} = \frac{40}{40 + 20} = \frac{40}{60} = 0.6 \overline{6} \approx 0.667$ $Recall_{dog} = \frac{40}{40 + 10} = \frac{40}{50} = 0.80$

Recall for each class (diagonal ÷ row sum):

Cat: row sum $= 30 + 15 + 5 = 50$ , recall $= \frac{30}{50} = 0.60$ .
Dog: $\frac{40}{50} = 0.80$ .
Bird: row sum $= 5 + 5 + 40 = 50$ , recall $= \frac{40}{50} = 0.80$ .

Macro recall — average the per-class recalls, each weighted equally: $Recall_{macro} = \frac{0.60 + 0.80 + 0.80}{3} = \frac{2.20}{3} \approx 0.7333$ See Multi-class Classification Metrics.

Exercise 4.2

For the same 3-class matrix, compute the micro-averaged recall and explain why it here equals overall accuracy.

Recall Solution

Micro pooling — add up all class-wise $T P$ and $F N$ first, then divide once:

$\sum T P =$ diagonal $= 30 + 40 + 40 = 110$ .
$\sum F N =$ all off-diagonal in each row (every misclassified sample is some class's FN) $= (15 + 5) + (10 + 0) + (5 + 5) = 40$ . $Recall_{micro} = \frac{\sum T P}{\sum T P + \sum F N} = \frac{110}{110 + 40} = \frac{110}{150} = 0.7 \overline{3} \approx 0.7333$ Overall accuracy $= \frac{diagonal}{total} = \frac{110}{150} = 0.7333$ . WHY equal: in single-label multi-class, every sample is exactly one class's positive. Summed $T P$ = all correct = diagonal, and $\sum T P + \sum F N$ = all samples. So micro-recall reduces to accuracy. (Note macro $\approx 0.733$ also coincidentally matches here only because every row sum is 50 — with imbalanced rows they'd differ.)

Level 5 — Mastery

Exercise 5.1

A cancer screen sees 1000 patients, 40 truly have cancer. A miss (FN) costs 100 units (a lethal error); a false alarm (FP) costs 1 unit (an extra biopsy). Model A: $T P = 32, F N = 8, F P = 60, T N = 900$ . Model B: $T P = 39, F N = 1, F P = 300, T N = 660$ . First compute each model's accuracy (to see the "same-looking" region), then decide which model is better using total cost, and separately by $F_{1}$ . Do the two criteria agree?

Recall Solution

First, accuracies — this is the "same-looking region." Both totals are $1000$ :

Model A: $\frac{T P + T N}{1000} = \frac{32 + 900}{1000} = \frac{932}{1000} = 0.932 (93.2%)$ .
Model B: $\frac{39 + 660}{1000} = \frac{699}{1000} = 0.699 (69.9%)$ .

So A actually looks far better on accuracy ( $93.2%$ vs $69.9%$ ) — a student glancing at accuracy would pick A immediately. That is exactly the trap; let the cost and $F_{1}$ tell a different story.

Total cost $= 100 \times F N + 1 \times F P$ :

Model A: $100 \times 8 + 60 = 800 + 60 = 860$ .
Model B: $100 \times 1 + 300 = 100 + 300 = 400$ . → Model B is cheaper (better) because it slashes the deadly misses, and each avoided miss is worth 100 cheap biopsies.

$F_{1}$ $= \frac{2 T P}{2 T P + F P + F N}$ :

Model A: $\frac{2 \times 32}{2 \times 32 + 60 + 8} = \frac{64}{132} \approx 0.485$ .
Model B: $\frac{2 \times 39}{2 \times 39 + 300 + 1} = \frac{78}{379} \approx 0.206$ . → $F_{1}$ prefers Model A.

They DISAGREE — and so does accuracy. Accuracy and $F_{1}$ both crown Model A, while the cost function crowns Model B. Accuracy is fooled by A's big $T N$ ; $F_{1}$ treats FP and FN symmetrically, so B's flood of false alarms tanks its score. But real-world cost weights a miss $100 \times$ heavier than a false alarm, flipping the decision to B. The takeaway: when the two error types carry very different real-world costs, neither accuracy nor $F_{1}$ is the right selection criterion — you must minimise an explicit cost function that encodes those weights. This is exactly Cost-Sensitive Learning and Model Selection Criteria.

Exercise 5.2

You must choose one operating threshold. At threshold $t_{1}$ : $(T P, F N, F P, T N) = (90, 10, 40, 860)$ . At $t_{2}$ : $(70, 30, 5, 995)$ — using the cost rule $cost = 5 \cdot F N + 1 \cdot F P$ . Which threshold wins, and sketch how this connects to picking a point on the ROC curve.

Recall Solution

Cost at each threshold:

$t_{1}$ : $5 \times 10 + 40 = 50 + 40 = 90$ .
$t_{2}$ : $5 \times 30 + 5 = 150 + 5 = 155$ . → $t_{1}$ wins (cost 90 < 155). Its higher recall (fewer FN) pays off because each FN costs $5 \times$ an FP.

ROC connection. The ROC curve plots TPR (= recall) on the vertical axis against FPR on the horizontal axis (both defined above the exercises). Each threshold is one point:

$t_{1}$ : $TPR = \frac{90}{100} = 0.90$ , $FPR = \frac{40}{40 + 860} = \frac{40}{900} \approx 0.044$ .
$t_{2}$ : $TPR = \frac{70}{100} = 0.70$ , $FPR = \frac{5}{5 + 995} = \frac{5}{1000} = 0.005$ . $t_{1}$ sits higher and to the right; $t_{2}$ lower and to the left. Sliding the threshold traces the whole ROC curve, and the cost ratio picks which point on that curve is optimal — the tangent point of an iso-cost line. See ROC Curve and AUC and Binary Classification Thresholds.

Figure 2 (described): ROC space, FPR on the x-axis and TPR on the y-axis, with a dashed diagonal "chance line." A black curve bows toward the top-left. Two operating points are marked: $t_{1}$ in red at (0.044, 0.90) — the cost-chosen winner, high and slightly right — and $t_{2}$ in black at (0.005, 0.70), lower and further left. If the image fails to load: the chosen threshold is the one with lower total cost, which here is $t_{1}$ .

Recall Self-test recap (reveal each answer)

Below, read the prompt, answer in your head, then check against the text after the dash.

Precision's denominator? — predicted positives, $T P + F P$ .
Recall's denominator? — actual positives, $T P + F N$ .
When is precision undefined? — when $T P + F P = 0$ (you never predicted positive).
When is recall undefined? — when $T P + F N = 0$ (no actual positives in the data).
FPR formula? — $\frac{F P}{F P + T N} = 1 - Specificity$ .
For class $i$ , where does its FN live? — the row off-diagonal (real $i$ 's sent elsewhere).
For class $i$ , where does its FP live? — the column off-diagonal (others stamped $i$ ).
Why does micro-recall equal accuracy in single-label multi-class? — summed TP = diagonal, and $T P + F N$ = all samples.
When is $F_{1}$ the wrong selection metric? — when FP and FN have very different real-world costs.