ROC curve and AUC: Exercises — AI-ML deep dive

Before we start, one shared picture of the two axes every ROC problem lives on.

Figure s01 — Anatomy of the ROC plot. A unit square: the horizontal axis is FPR (false-positive rate, 0 to 1), the vertical axis is TPR (true-positive rate, 0 to 1). A dashed white diagonal marks the random classifier (AUC = 0.5); a blue curve bulging toward the top-left marks a good classifier. Yellow dots mark the two fixed corners: $(0, 0)$ where the threshold is 1 (nothing called positive) and $(1, 1)$ where the threshold is 0 (everything called positive).

Recall The two rates in one breath

TPR (up the page) ::: Of all the truly-positive examples, the fraction we correctly flagged: $T P / (T P + F N)$ . FPR (across the page) ::: Of all the truly-negative examples, the fraction we wrongly flagged: $F P / (F P + T N)$ . The bottom-left corner $(0, 0)$ means ::: threshold so high nothing is called positive. The top-right corner $(1, 1)$ means ::: threshold so low everything is called positive.

Level 1 — Recognition

L1.1

A classifier's ROC curve lies exactly on the diagonal from $(0, 0)$ to $(1, 1)$ . What is its AUC, and what does this say about the model?

Recall Solution

The area under a straight diagonal of a unit square is a triangle: $\frac{1}{2} \cdot 1 \cdot 1 = 0.5$ . So $AUC = 0.5$ . This is a random classifier — it ranks a positive above a negative exactly half the time, no better than a coin flip.

L1.2

For a threshold $t$ , a model gives $T P = 30$ , $F N = 10$ , $F P = 5$ , $T N = 55$ . Compute TPR and FPR.

Recall Solution

$TPR = \frac{T P}{T P + F N} = \frac{30}{30 + 10} = \frac{30}{40} = 0.75$ $FPR = \frac{F P}{F P + T N} = \frac{5}{5 + 55} = \frac{5}{60} = \frac{1}{12} \approx 0.0833$ This is a single point $(0.083, 0.75)$ on the ROC curve — one dot for one threshold.

L1.3

Rank these AUC values from best to worst discrimination: $0.50, 0.91, 0.30, 0.72$ .

Recall Solution

Intended answer (raw AUC, as reported): $0.91 > 0.72 > 0.50 > 0.30$ . Higher AUC = better as-is; $0.30$ sits below the random baseline $0.50$ , so as reported it is the worst.

A subtlety worth knowing: an AUC below 0.5 is worse than random but flippable — a model at $0.30$ becomes $1 - 0.30 = 0.70$ if you invert every prediction. So its usable discrimination is $0.70$ , which would rank third:

Raw ranking (the intended answer): $0.91, 0.72, 0.50, 0.30$ .
Usable ranking (after allowing a flip): $0.91, 0.72, 0.70, 0.50$ . Unless the question says "you may flip," give the raw ranking: $0.91, 0.72, 0.50, 0.30$ .

Level 2 — Application

L2.1

Six examples, scores and labels below. Compute $n_{+}$ , $n_{-}$ , and the ROC point at threshold $t = 0.55$ (predict positive if $s_{i} \geq t$ ).

Ex	Score	Label
A	0.90	1
B	0.70	0
C	0.60	1
D	0.50	1
E	0.40	0
F	0.20	0

Recall Solution

$n_{+} = 3$ (A, C, D), $n_{-} = 3$ (B, E, F). At $t = 0.55$ , predicted-positive = scores $\geq 0.55$ = {A (0.90), B (0.70), C (0.60)}. Among those: A→label 1 (TP), C→label 1 (TP), B→label 0 (FP). So $T P = 2$ , $F P = 1$ . $TPR = \frac{2}{3} \approx 0.667, FPR = \frac{1}{3} \approx 0.333.$ Point on the curve: $(0.333, 0.667)$ .

L2.2

Using the Mann–Whitney U formula, compute the AUC for the six examples of L2.1. Here $s_{i}$ is the score of example $i$ and $y_{i}$ its true label, and $1 [\cdot]$ is the indicator function (defined in the Notation box above — $1$ if the bracketed condition holds, else $0$ ): $AUC = \frac{1}{n _{+} n _{-}} \sum_{i : y_{i} = 1} \sum_{j : y_{j} = 0} (1 [s_{i} > s_{j}] + 0.5 1 [s_{i} = s_{j}]) .$

Recall Solution

The outer sum runs over positives (label $y_{i} = 1$ ), the inner over negatives (label $y_{j} = 0$ ). Each term counts $1$ if the positive out-scores the negative, $0.5$ for a tie, $0$ otherwise. Positives: A(0.90), C(0.60), D(0.50). Negatives: B(0.70), E(0.40), F(0.20). Count, for each positive, how many negatives it out-scores (no ties here):

A = 0.90: beats B(0.70), E(0.40), F(0.20) → 3
C = 0.60: beats E(0.40), F(0.20), loses to B(0.70) → 2
D = 0.50: beats E(0.40), F(0.20), loses to B(0.70) → 2

Total correctly-ordered pairs $= 3 + 2 + 2 = 7$ out of $n_{+} n_{-} = 3 \times 3 = 9$ . $AUC = \frac{7}{9} \approx 0.778.$

L2.3

A model always outputs the same score $0.5$ for every example (positives and negatives alike). What is its AUC? Use the tie term in the Mann–Whitney formula.

Recall Solution

Every positive–negative pair is a tie, so each contributes $0.5$ . $AUC = \frac{1}{n _{+} n _{-}} \sum (0.5) = \frac{0.5 \cdot n _{+} n _{-}}{n _{+} n _{-}} = 0.5.$ A constant model has no ranking power → exactly the random baseline. This confirms why we assign $0.5$ to ties: it keeps a no-information model honestly at $0.5$ .

Level 3 — Analysis

L3.1

Take Worked Example 1 from the parent note (8 examples, AUC = 0.6875). Two of the ROC points are $(0.25, 0.5)$ and $(0.25, 0.75)$ . Explain geometrically why the curve moves straight up (FPR constant, TPR rising) between these two points, and which example causes it.

Figure s02 — Staircase steps. The 9-vertex ROC path of the parent's Worked Example 1, drawn in blue with dotted markers over the dashed random diagonal. A thick green segment highlights the up-step from $(0.25, 0.5)$ to $(0.25, 0.75)$ — caused by crossing a positive (example D), so TPR rises by $1/4$ while FPR is unchanged. A thick red segment highlights a right-step from $(0.25, 0.75)$ to $(0.5, 0.75)$ — caused by crossing a negative (example E), so FPR rises by $1/4$ while TPR is unchanged.

Recall Solution

Sorted scores: A(0.9,1), B(0.8,1), C(0.7,0), D(0.6,1), E(0.55,0), F(0.5,1), G(0.3,0), H(0.1,0). Lowering the threshold past D (score 0.6, label 1) turns one more true positive on. A true positive raises $T P$ (so TPR climbs) but leaves $F P$ untouched (so FPR stays put). Hence a vertical move up: $(0.25, 0.5) \to (0.25, 0.75)$ . The rule: crossing a positive example → step up (height $1/ n_{+}$ ); crossing a negative → step right (width $1/ n_{-}$ ). Every ROC curve is a staircase of up-steps and right-steps.

L3.2

Same dataset. What is the maximum achievable TPR at FPR = 0 (i.e. before any false positive)? Which examples give it?

Recall Solution

Read the sorted list from the top until the first negative appears. The top two, A and B, are both positives; the third, C, is the first negative. So we catch 2 of 4 positives with zero false positives → TPR $= 2/4 = 0.5$ at FPR $= 0$ . Geometrically this is the highest point the curve reaches while still hugging the left edge $x = 0$ .

L3.3

Model P and Model Q have the same AUC = 0.80 but their ROC curves cross. Model P bulges toward the top-left at low FPR; Model Q bulges near high TPR at high FPR. You need a low-false-alarm operating point. Which model do you pick, and why is AUC alone not enough?

Recall Solution

Pick Model P. AUC is one number summarising the whole curve; two curves with equal area can trade strength in different regions. At low FPR (the region you care about), P sits higher — more true positives caught per false alarm. Q's area is "earned" out at high FPR, useless to you. Lesson: when the operating region is fixed, compare the curves locally (partial AUC or the TPR at your target FPR), not the global AUC.

Level 4 — Synthesis

L4.1

A fraud dataset has $n_{+} = 100$ frauds and $n_{-} = 9900$ legit transactions. At one threshold, $T P = 80$ , $F P = 495$ . (a) Compute TPR and FPR. (b) Compute Precision $= T P / (T P + F P)$ . (c) Explain why ROC "looks fine" but precision reveals the problem.

Recall Solution

(a) $TPR = 80/100 = 0.80$ . $FPR = 495/9900 = 0.05$ . (b) $Precision = \frac{80}{80 + 495} = \frac{80}{575} \approx 0.139$ . (c) On the ROC axes this point $(0.05, 0.80)$ looks excellent — high catch rate, tiny FPR. But an FPR of $0.05$ on 9900 negatives is $495$ false alarms, dwarfing the $80$ true frauds. Precision $\approx 14%$ : 6 out of 7 flagged transactions are innocent. ROC hides this because FPR divides by the huge negative count. This is precisely why the parent note's Mistake 2 recommends Precision-Recall Curve on imbalanced data.

L4.2

Combine ranking and area. Ten examples produce the ROC staircase with vertices $(0, 0), (0, 0.2), (0.2, 0.2), (0.2, 0.6), (0.4, 0.6), (0.4, 1.0), (1.0, 1.0) .$ Compute the AUC by the trapezoid sum $AUC = \sum_{i} \frac{1}{2} (FPR_{i + 1} - FPR_{i}) (TPR_{i} + TPR_{i + 1})$ .

Recall Solution

Only segments where FPR increases contribute area (vertical segments have zero width):

$(0.2, 0.2) \to (0.2, 0.6)$ : width $0$ → contributes $0$ .
$(0, 0.2) \to (0.2, 0.2)$ : $\frac{1}{2} (0.2) (0.2 + 0.2) = \frac{1}{2} (0.2) (0.4) = 0.04$ .
$(0.2, 0.6) \to (0.4, 0.6)$ : $\frac{1}{2} (0.2) (0.6 + 0.6) = \frac{1}{2} (0.2) (1.2) = 0.12$ .
$(0.4, 1.0) \to (1.0, 1.0)$ : $\frac{1}{2} (0.6) (1.0 + 1.0) = \frac{1}{2} (0.6) (2.0) = 0.60$ .

$AUC = 0.04 + 0.12 + 0.60 = 0.76.$

L4.3

You have two thresholds giving points $(0.1, 0.7)$ and $(0.3, 0.9)$ . A downstream system needs Specificity $\geq 0.85$ . Recall Specificity $= 1 - FPR$ . Which of the two thresholds is admissible, and what is its Sensitivity?

Recall Solution

Specificity $= 1 - FPR$ .

Point $(0.1, 0.7)$ : Specificity $= 1 - 0.1 = 0.90 \geq 0.85$ ✓. Sensitivity (=TPR) $= 0.70$ .
Point $(0.3, 0.9)$ : Specificity $= 1 - 0.3 = 0.70 < 0.85$ ✗. Only the first threshold satisfies the constraint; it delivers Sensitivity 0.70 (see Sensitivity and Specificity). You trade some recall to meet the specificity floor.

Level 5 — Mastery

L5.1 (Degenerate curve)

A dataset has $n_{+} = 5$ positives but zero negatives ( $n_{-} = 0$ ). What happens to FPR, and can you draw an ROC curve?

Recall Solution

$FPR = F P / (F P + T N) = F P / n_{-}$ requires dividing by $n_{-} = 0$ → undefined. With no negatives there is no x-axis to sweep; the ROC curve cannot be defined. AUC is likewise undefined. This is a data problem, not a model problem: ROC/AUC require both classes present. (Same collapse happens to TPR if $n_{+} = 0$ .)

L5.2 (Perfect separation)

Positives all score above $0.6$ and negatives all score below $0.4$ . Sketch the ROC path and give the AUC. Explain via the Mann–Whitney interpretation.

Figure s03 — Perfect separation. The ROC path $(0, 0) \to (0, 1) \to (1, 1)$ drawn in green, hugging the left edge then the top edge, with the enclosed unit square lightly shaded green (area = 1) over the dashed random diagonal. A yellow arrow marks that all positives are caught before any false positive appears, giving AUC = 1.

Recall Solution

As you lower the threshold from $1$ : you first sweep past all positives (curve climbs straight up the left edge to $(0, 1)$ ) before touching a single negative, because the lowest positive still beats every negative. Then, continuing down, you sweep the negatives, moving right along the top edge to $(1, 1)$ . Path: $(0, 0) \to (0, 1) \to (1, 1)$ . This traces the top-left corner → area of the entire unit square $= 1$ . $AUC = 1.$ Mann–Whitney check: every positive out-scores every negative, so all $n_{+} n_{-}$ pairs are correctly ordered → $\frac{n _{+} n _{-}}{n _{+} n _{-}} = 1$ . Perfect separation ⇔ AUC $= 1$ .

L5.3 (Flipped model)

A model has AUC $= 0.20$ . (a) What single operation makes it useful? (b) What is the resulting AUC? (c) Why does this operation reflect the ROC curve about a specific line?

Recall Solution

(a) Negate the scores (replace each $s_{i}$ by $- s_{i}$ , or equivalently swap the two predicted labels): every ranking reverses. (b) Reversing the order of every positive–negative pair turns each correctly-ordered pair into an incorrectly-ordered one and vice-versa, so the fraction correct becomes $1$ minus the old fraction: $AUC = 1 - 0.20 = 0.80$ . (c) On the ROC plot, negating scores maps each point $(FPR, TPR)$ to $(1 - FPR, 1 - TPR)$ — a 180° rotation about the centre $(0.5, 0.5)$ (equivalently, reflection through that centre point). A curve that bulged below the diagonal now bulges above it. That symmetry is exactly why AUC and $1 - AUC$ pair up.

L5.4 (Threshold count)

With $n$ examples all having distinct scores, how many distinct ROC vertices (including $(0, 0)$ and $(1, 1)$ ) can the curve have at most, and why?

Recall Solution

Sweeping the threshold below each distinct score produces one new vertex, plus the starting corner $(0, 0)$ . With $n$ distinct scores you get $n$ crossings, giving $n + 1$ vertices total (from $(0, 0)$ to $(1, 1)$ ). For $n = 8$ (parent Worked Example 1): $8 + 1 = 9$ vertices — matching the 9-point path listed there. Each vertex is either an up-step (positive crossed) or a right-step (negative crossed).

Recall Self-test checklist (reveal after finishing)

Diagonal AUC ::: $0.5$ , a random classifier. AUC below 0.5 fix ::: negate scores → new AUC $= 1 - AUC$ . Up-step vs right-step ::: crossing a positive steps up $1/ n_{+}$ ; crossing a negative steps right $1/ n_{-}$ . ROC undefined when ::: a class is empty ( $n_{+} = 0$ or $n_{-} = 0$ ). Ties in Mann–Whitney ::: count as $0.5$ each. Why ROC misleads on imbalance ::: FPR divides by the huge negative count, hiding large false-alarm counts.