Precision-recall tradeoff and curves: Exercises

This is a self-testing worksheet for the parent topic. Every problem has a fully worked solution hidden inside a collapsible callout — try first, then reveal. Exercises climb from L1 Recognition (do you know the words?) up to L5 Mastery (can you invent and defend a decision?).

Before we start, one shared reminder of the four counts we will use everywhere. Picture sorting every prediction into one of four boxes — look at the figure below as you read the definitions:

Figure — Precision-recall tradeoff and curves

Definition The four boxes (say them out loud, tracing the figure)

In the figure, the two mint boxes are the answers we got right, the two coral boxes are our two kinds of error.

$T P$ (true positive) — top-left mint box: we said yes, and it really was yes. A correct catch.
$F N$ (false negative) — top-right coral box: we said no, but it was yes. A miss.
$F P$ (false positive) — bottom-left coral box: we said yes, but it was no. A false alarm.
$T N$ (true negative) — bottom-right mint box: we said no, and it really was no. A correct pass.

$Precision = \frac{T P}{T P + F P} Recall = \frac{T P}{T P + F N}$

Follow the lavender vertical arrow in the figure: precision reads down the "we said yes" column ( $T P$ over $T P + F P$ ). Follow the coral horizontal arrow: recall reads across the "actually yes" row ( $T P$ over $T P + F N$ ). That is the whole game. See 2.6.01-Confusion-matrix-and-basic-metrics if any box feels shaky.

Definition Edge cases: when precision or recall is

undefined Both metrics are fractions, so we must ask what happens when a denominator hits zero.

$T P + F P = 0$ (we predicted no positives at all — e.g. $τ$ set above every score). Precision $= \frac{0}{0}$ is undefined. Conventionally we either report it as $0$ or leave the point off the curve; sklearn sets it to $0$ with a warning. Meaning: "we never said yes, so 'how often were we right when we said yes?' has no answer."
$T P + F N = 0$ (there are no actual positives in the data). Recall $= \frac{0}{0}$ is undefined — you literally have no positives to catch, so recall is meaningless; you need a test set that contains some positives.
A threshold so low that everything is predicted positive gives $F N = 0$ , so recall $= 1$ and precision collapses to the prevalence. Keep these in mind: a "no positives predicted" threshold is a legal but degenerate endpoint of every sweep.

Level 1 — Recognition

Goal: read the counts and plug into the definitions. No threshold sweeping yet.

Exercise 1.1 (L1)

A spam filter processes 200 emails. It flags 50 as spam. Of those 50, 40 really are spam. There were 60 spam emails in total. Compute precision and recall.

Recall Solution 1.1

"Flagged as spam" = predicted positive = 50 emails. Of those, 40 are correct, so $T P = 40, F P = 50 - 40 = 10.$ Total actual spam = 60, we caught 40, so we missed $F N = 60 - 40 = 20.$ $Precision = \frac{40}{40 + 10} = \frac{40}{50} === 0.80 ==$ $Recall = \frac{40}{40 + 20} = \frac{40}{60} \approx== 0.667 ==$ Reading it back: when the filter shouts "spam" it is right $80%$ of the time (precision), and it catches two-thirds of all real spam (recall).

Exercise 1.2 (L1)

For the same filter, which metric would you watch if your biggest fear is a real invoice landing in the spam folder (a false positive that hurts the user)?

Recall Solution 1.2

A real email wrongly sent to spam is a false positive ( $F P$ ). The metric whose denominator is punished by $F P$ is $Precision = \frac{T P}{T P + F P} .$ So you watch precision. Higher precision = fewer false alarms = fewer lost invoices.

Level 2 — Application

Goal: pick a threshold, build the confusion counts yourself, compute both metrics.

Exercise 2.1 (L2)

A model scores 8 patients (higher score = more likely diseased). Sorted:

Patient	Score	True label
A	0.92	Sick
B	0.81	Sick
C	0.70	Healthy
D	0.60	Sick
E	0.44	Healthy
F	0.35	Sick
G	0.22	Healthy
H	0.10	Healthy

Predict "sick" when score $s \geq τ = 0.50$ . Compute precision and recall. Total sick = 4.

Recall Solution 2.1

Predicted positive (score $\geq 0.50$ ): A, B, C, D.

A sick ✅, B sick ✅, C healthy ❌, D sick ✅.
So $T P = 3$ (A,B,D), $F P = 1$ (C).
Missed sick people: F (0.35) is below $τ$ , so $F N = 1$ . $Precision = \frac{3}{3 + 1} === 0.75 ==, Recall = \frac{3}{3 + 1} === 0.75 == .$

Exercise 2.2 (L2)

Same table. Now lower the threshold to $τ = 0.30$ . Recompute precision and recall, and state in one sentence which way each moved and why.

Recall Solution 2.2

Predicted positive (score $\geq 0.30$ ): A, B, C, D, E, F.

Sick among them: A, B, D, F → $T P = 4$ .
Healthy among them: C, E → $F P = 2$ .
Sick people missed: none left ( $F N = 0$ ). $Precision = \frac{4}{4 + 2} \approx== 0.667 ==, Recall = \frac{4}{4 + 0} === 1.00 == .$ What moved: recall rose $0.75 \to 1.00$ (we now catch every sick patient), precision fell $0.75 \to 0.667$ (we let in one extra healthy person, E). Lowering $τ$ traded precision for recall — the tradeoff in action. See 3.2.04-Decision-thresholds-and-calibration.

Level 3 — Analysis

Goal: reason about whole curves, baselines, and why ROC and PR disagree.

Exercise 3.1 (L3)

A dataset has 1000 samples: 20 positive, 980 negative. A model flags 100 as positive; of those, $T P = 16$ . Compute (a) precision, (b) recall, (c) FPR, and (d) explain why the ROC picture looks far rosier than the PR picture.

Recall Solution 3.1

Predicted positive = 100, correct = 16, so $F P = 100 - 16 = 84$ . Actual positives = 20, caught 16, so $F N = 20 - 16 = 4$ . True negatives $T N = 980 - 84 = 896$ .

(a) $Precision = \frac{16}{16 + 84} === 0.16 == .$
(b) $Recall (=TPR) = \frac{16}{16 + 4} === 0.80 == .$
(c) $FPR = \frac{F P}{F P + T N} = \frac{84}{84 + 896} = \frac{84}{980} \approx== 0.0857 == .$
(d) ROC plots recall ( $0.80$ ) against FPR ( $0.086$ ) — that looks like a great point (top-left-ish). But FPR divides by the huge $T N = 896$ , so 84 false alarms barely register. PR plots precision ( $0.16$ ), which ignores $T N$ entirely and sees those 84 false alarms clearly. On imbalanced data the PR view is honest, the ROC view is flattering. Compare with 2.6.07-ROC-curves-and-AUC and 2.6.12-Class-imbalance-handling.

Exercise 3.2 (L3)

A fraud dataset is 3% fraud. Your model gets AUC-PR $= 0.24$ . Is that good? Compute the no-skill baseline and the lift over it.

Recall Solution 3.2

The PR no-skill baseline equals the positive-class fraction (a coin-flip classifier's expected precision is just the prevalence of positives): $baseline = 0.03.$ $lift = \frac{0.24}{0.03} === 8 == .$ So AUC-PR $= 0.24$ is 8× better than random — genuinely good here, even though $0.24$ sounds low in isolation. Never compare AUC-PR to $0.5$ ; compare it to prevalence.

Exercise 3.3 (L3)

Given three PR operating points sorted by increasing recall — $(r, p) = (0.2, 0.90), (0.5, 0.70), (0.9, 0.40)$ — estimate AUC-PR by the trapezoidal rule.

Recall Solution 3.3

Why the trapezoid rule here? We only know precision at a handful of thresholds — the true curve $p (r)$ between them is unmeasured. We need a way to guess the area under those unknown gaps. The simplest honest guess is to connect each pair of known points with a straight line and measure the area of the resulting slab. That slab is a trapezoid (two vertical sides of heights $p_{i}, p_{i + 1}$ , a horizontal base of width $r_{i + 1} - r_{i}$ ), and a trapezoid's area is (average of the two heights) × width. This assumes precision varies roughly linearly between adjacent points — a reasonable, unbiased middle-ground compared to assuming it stays flat (over/under-estimates). So: $area of one slab = \frac{p _{i} + p _{i + 1}}{2} (r_{i + 1} - r_{i}) .$ First segment ( $0.2 \to 0.5$ ): $\frac{0.90 + 0.70}{2} (0.5 - 0.2) = 0.80 \times 0.3 = 0.24.$ Second segment ( $0.5 \to 0.9$ ): $\frac{0.70 + 0.40}{2} (0.9 - 0.5) = 0.55 \times 0.4 = 0.22.$ $AUC-PR \approx 0.24 + 0.22 === 0.46 == .$ The purple shaded region in the figure below is these two trapezoids stacked side by side.

Look at the shaded area — each vertical dotted line marks a trapezoid boundary, and the mint dashed line shows the no-skill floor we compared against in Exercise 3.2:

Definition PR-curve boundary behaviour (the two endpoints)

The trapezoid answer depends on what we do at the edges of the recall axis, so name them explicitly.

At recall $= 0$ (the strictest usable threshold, only the top-scoring items are positive): precision is whatever those very top predictions achieve — often near $1$ , but if the single highest-scored item is wrong, it can be $0$ . sklearn anchors the curve at precision $= 1$ , recall $= 0$ by convention so the curve starts at the top-left corner.
At recall $= 1$ (threshold so low all positives are caught): precision equals the prevalence (fraction positive), because you've predicted almost everything positive.
Interpolation choice matters: the plain trapezoid rule (straight lines, used above) can over-estimate AUC-PR because the real PR curve is often "sawtoothed." That is why sklearn's average_precision_score uses a step (right-constant) interpolation instead: $AP = \sum_{i} (r_{i} - r_{i - 1}) p_{i}$ , which avoids optimistic diagonal shortcuts. Our trapezoid figure of $0.46$ is a teaching approximation; report average precision for production numbers.

Level 4 — Synthesis

Goal: combine metrics, choose thresholds under a stated goal, connect to F-score.

Exercise 4.1 (L4)

Using Exercise 2.1's model, you must catch at least 90% of sick patients (recall $\geq 0.90$ ) while keeping precision as high as possible. Which threshold from ${0.85, 0.50, 0.40, 0.30, 0.20}$ do you pick? Show the recall/precision at your choice.

Recall Solution 4.1

Sweep thresholds on the 8-patient table (4 sick: A,B,D,F).

$τ = 0.85$ : positives {A}. $T P = 1, F P = 0, F N = 3$ . Recall $= 0.25$ . Fails recall.
$τ = 0.50$ : positives {A,B,C,D}. $T P = 3, F P = 1, F N = 1$ . Recall $= 0.75$ . Fails recall.
$τ = 0.40$ : positives {A,B,C,D,E}. Sick caught A,B,D → $T P = 3$ ; F(0.35) still out → $F N = 1$ . Recall $= 0.75$ . Fails recall.
$τ = 0.30$ : positives {A,B,C,D,E,F}. $T P = 4, F P = 2, F N = 0$ . Recall $= 1.00$ ✅, Precision $= \frac{4}{6} \approx 0.667$ .
$τ = 0.20$ : adds G(healthy). $T P = 4, F P = 3, F N = 0$ . Recall $= 1.00$ , Precision $= \frac{4}{7} \approx 0.571$ .

The only thresholds hitting recall $\geq 0.90$ are $0.30$ and $0.20$ . Between them, $τ = 0.30$ gives higher precision. Pick $τ = 0.30$ : recall $=== 1.00 ==$ , precision $\approx== 0.667 ==$ .

Exercise 4.2 (L4)

At the two recall-satisfying thresholds above ( $0.30$ and $0.20$ ), compute the F1 score $F_{1} = \frac{2 p r}{p + r}$ and confirm it prefers the same threshold.

Recall Solution 4.2

$F_{1}$ is the harmonic mean of precision and recall — it punishes the lower of the two, so it favours balanced points. See 2.6.09-F1-score-and-Fbeta-score. At $τ = 0.30$ : $p = \frac{4}{6} = 0.6 \overline{6}, r = 1$ : $F_{1} = \frac{2 ( 0.6667 ) ( 1 )}{0.6667 + 1} = \frac{1.3333}{1.6667} \approx== 0.80 == .$ At $τ = 0.20$ : $p = \frac{4}{7} = 0.5714, r = 1$ : $F_{1} = \frac{2 ( 0.5714 ) ( 1 )}{0.5714 + 1} = \frac{1.1429}{1.5714} \approx== 0.727 == .$ $F_{1}$ is higher at $τ = 0.30$ ( $0.80 > 0.727$ ), agreeing with our precision-based choice.

Level 5 — Mastery

Goal: design and defend a decision where the "right" metric depends on real-world cost.

Exercise 5.1 (L5)

Two fraud models on the same 2%-fraud test set:

Model X: AUC-ROC $= 0.95$ , AUC-PR $= 0.30$ .
Model Y: AUC-ROC $= 0.90$ , AUC-PR $= 0.55$ .

Each missed fraud costs $1000 (a $F N$ ); each false alarm costs $5 of analyst review (a $F P$ ). Which model do you ship, and which curve did you trust — and why?

Recall Solution 5.1

Step 1 — which curve is trustworthy here? The set is $2%$ positive: heavily imbalanced. ROC's FPR hides false positives behind a giant $T N$ , so AUC-ROC over-praises both models. PR is the honest curve (see Exercise 3.1's logic). So compare AUC-PR: $0.30$ vs $0.55$ . Step 2 — baseline check. PR baseline $= 0.02$ . Model X lift $= 0.30/0.02 === 15 ==$ ; Model Y lift $= 0.55/0.02 === 27.5 ==$ . Both beat random, Y far more. Step 3 — cost lens. Missing fraud costs $1000, a false alarm only $5 — a 200:1 ratio. High precision (few false alarms) is cheap; high recall (few misses) is what saves money. Model Y's much larger AUC-PR means it sustains higher precision and recall across thresholds, so at any operating point matching X's recall it produces fewer false alarms, and at any precision it catches more fraud. Decision: ship Model Y, trusting AUC-PR over AUC-ROC because the class imbalance makes ROC optimistic. See 4.1.08-Cost-sensitive-learning.

Exercise 5.2 (L5)

For your chosen Model Y, suppose at some threshold: on $10, 000$ transactions ( $200$ fraud) you get $T P = 150, F P = 300$ . Compute the total expected cost with the $1000/$5 figures above, and compare it to the "flag nothing" baseline.

Recall Solution 5.2

Missed fraud: $F N = 200 - 150 = 50$ . False alarms: $F P = 300$ . $Cost = 1000 \times F N + 5 \times F P = 1000 (50) + 5 (300) = 50000 + 1500 = $ == 51500 == .$ "Flag nothing" baseline: every fraud is missed, $F N = 200$ , $F P = 0$ : $Cost_{none} = 1000 \times 200 = $ == 200000 == .$ Model Y saves $200000 - 51500 = \$ 148500 $v er s u s d o in g n o t hin g — t h e f a l se - a l a r m cos t ($ $1500$) is trivial next to the fraud it prevents. This is why recall dominates the decision here.

Recall Self-check summary (reveal after finishing)

Metric divides by the "yes" you produced ::: Precision uses $T P + F P$ Metric divides by the "yes" that truly existed ::: Recall uses $T P + F N$ When precision is undefined ::: when $T P + F P = 0$ (no positives predicted at all) When recall is undefined ::: when $T P + F N = 0$ (no actual positives in the data) PR curve's random baseline ::: the fraction of positive samples (the prevalence) Why ROC flatters imbalanced data ::: FPR's denominator $F P + T N$ is dominated by the huge $T N$ Why the trapezoid rule fits AUC-PR ::: it linearly connects known points and averages the two heights times the width Precision at recall = 1 ::: it drops to the prevalence, because nearly everything is predicted positive Constrained threshold choice ::: fix one metric as a floor, optimize the other among feasible thresholds Right metric under heavy imbalance and cost asymmetry ::: AUC-PR and expected cost, not AUC-ROC