ROC curve and AUC: Worked examples — AI-ML deep dive

This page is the "worked examples" companion to the ROC & AUC topic note. The parent note showed you what a ROC curve and AUC are. Here we throw every kind of situation at them — perfect models, useless models, backwards models, ties, tiny datasets, hugely imbalanced datasets, and an exam-style trap — and grind each one through to a number you can verify.

Before we start, a couple of reminders in plain words so no symbol sneaks in undefined:

The scenario matrix

Every ROC/AUC problem you will ever meet falls into one of these cells. The examples below are each tagged with the cell(s) they cover.

Cell	Case class	What makes it special	Example
C1	Perfect separation	Every positive scores above every negative	Ex 1
C2	Worse-than-random	AUC $< 0.5$ ; scores are backwards	Ex 2
C3	Above-diagonal staircase	Alternating labels, signal still present	Ex 3
C4	Tied scores	Two examples share a score → the 0.5 rule	Ex 4
C5	Degenerate: one class empty	$n_{+} = 0$ or $n_{-} = 0$ → rate undefined	Ex 5
C6	Both methods agree	Trapezoid area = Mann–Whitney pair count	Ex 6
C7	Severe imbalance	ROC-AUC looks great but false alarms flood you	Ex 7
C8	Real-world / cost twist	Pick a threshold under a cost rule	Ex 8
C9	Exam trap	High accuracy, useless AUC	Ex 9

Two tools reappear, so let me name them once, plainly:

Intuition The two ways to get AUC — same number, two stories

Trapezoid story (geometry): draw the staircase ROC curve, add up the little trapezoids under it. AUC = area. Pairing story (counting): take every pair (one real positive, one real negative). Count what fraction of pairs the positive out-scored the negative. Ties count as half a point. That fraction IS the AUC. This is the Mann–Whitney U formula: $AUC = \frac{1}{n _{+} n _{-}} \sum_{i : y_{i} = 1} \sum_{j : y_{j} = 0} (1 [s_{i} > s_{j}] + \frac{1}{2} 1 [s_{i} = s_{j}])$ Recall $s_{i}$ is a score and $y_{i}$ its true label. So $\sum_{i : y_{i} = 1}$ means "sum over every real positive $i$ " and $\sum_{j : y_{j} = 0}$ means "sum over every real negative $j$ ." $1 [\cdot]$ means "1 if the thing inside is true, else 0." So we look at every (positive $i$ , negative $j$ ) pair and score 1 if $s_{i} > s_{j}$ , a half if they tie. Why two tools? The geometry tool is what you plot and eyeball; the pairing tool is faster by hand and explains why AUC is a probability. Ex 6 proves they land on the same value.

Definition The trapezoid area rule (used all over this page)

The ROC curve is a chain of points $(FPR_{0}, TPR_{0}), (FPR_{1}, TPR_{1}), \dots$ from left to right. The area under one straight segment from point $k$ to point $k + 1$ is a trapezoid: base width $= FPR_{k + 1} - FPR_{k}$ , and its two vertical sides have heights $TPR_{k}$ and $TPR_{k + 1}$ . A trapezoid's area is (width) × (average of the two heights): $area_{k} = (FPR_{k + 1} - FPR_{k}) \cdot \frac{TPR _{k} + TPR _{k + 1}}{2}$ Add them all up for the total AUC. Why this and not a rectangle? A rectangle would assume the curve is flat across the step; the trapezoid correctly follows the slanted line the ROC actually draws. Notice: when the width is $0$ (a purely vertical rise, FPR unchanged) that trapezoid contributes exactly 0 — a vertical line encloses no area to its left.

Example 1 — Perfect separation (Cell C1)

Worked example All positives outrank all negatives

Four examples. Scores/labels:

Ex	Score	Label
P1	0.90	1
P2	0.80	1
N1	0.40	0
N2	0.20	0

$n_{+} = 2, n_{-} = 2$ . Find the ROC curve and AUC.

Forecast: Guess before computing — will AUC be exactly 1?

Step 1. Sort by score descending: P1, P2, N1, N2. Why this step? Sweeping the threshold from high to low visits scores in exactly this order; each score crossed adds one example to the "predicted positive" pile.

Step 2. Walk the threshold down. Each time we cross a positive, TPR climbs; each time we cross a negative, FPR climbs.

Cross P1 → $T P = 1$ : point $(0, 0.5)$
Cross P2 → $T P = 2$ : point $(0, 1.0)$
Cross N1 → $F P = 1$ : point $(0.5, 1.0)$
Cross N2 → $F P = 2$ : point $(1.0, 1.0)$

Why this step? Because all positives come first, TPR reaches 1 before any FPR appears. The curve hugs the top-left corner.

Step 3. Apply the trapezoid rule. The segments (0,0)→(0,0.5)→(0,1.0) have width 0 → area 0. The segment (0,1.0)→(0.5,1.0) has width $0.5$ , heights $1.0$ and $1.0$ : area $= 0.5 \cdot \frac{1.0 + 1.0}{2} = 0.5$ . Segment (0.5,1.0)→(1.0,1.0): width $0.5$ , heights $1.0, 1.0$ : area $= 0.5$ . Total $= 0.5 + 0.5 =$ 1.0. Why this step? We literally add the little areas under each straight piece; the two vertical rises at FPR=0 add nothing, and the two flat pieces at TPR=1 fill the whole square.

Verify: Pairing story — pairs are (P1,N1),(P1,N2),(P2,N1),(P2,N2). Positive outscores negative in all 4. $AUC = 4/4 = 1.0$ . ✓ Matches geometry.

Example 2 — Worse than random (Cell C2)

Worked example Scores are backwards

Take Example 1 but flip the labels' scores — negatives now score high:

Ex	Score	Label
N1	0.90	0
N2	0.80	0
P1	0.40	1
P2	0.20	1

Forecast: AUC above or below 0.5?

Step 1. Sort descending: N1, N2, P1, P2. Why this step? Same reason — threshold sweep visits highest scores first, and here those are all negatives.

Step 2. Walk down:

Cross N1 → $F P = 1$ : $(0.5, 0)$
Cross N2 → $F P = 2$ : $(1.0, 0)$
Cross P1 → $T P = 1$ : $(1.0, 0.5)$
Cross P2 → $T P = 2$ : $(1.0, 1.0)$

The curve runs along the bottom first (FPR climbs while TPR stays 0), hugging the bottom-right — the mirror image of Ex 1.

Step 3. Trapezoid rule. Segments along the bottom: (0,0)→(0.5,0)→(1.0,0) have TPR heights $0$ and $0$ , so each area $=$ width $\times \frac{0 + 0}{2} = 0$ . The vertical segments at FPR=1.0 have width 0 → area 0. Total area $=$ 0.0. Why this step? Every piece of the curve either sits at TPR=0 (no height, no area) or is vertical (no width, no area) — nothing lies under it.

Verify: Pairing — in every one of the 4 (positive, negative) pairs the negative scored higher, so the positive-outscores-negative count is 0. $AUC = 0/4 = 0$ . ✓ Sanity: AUC $< 0.5$ means "flip your predictions." Flipping gives AUC $= 1$ — exactly Example 1. Consistent. ✓

Example 3 — Above-diagonal staircase (Cell C3)

Worked example Alternating labels — but is the signal really gone?

Ex	Score	Label
A	0.80	1
B	0.60	0
C	0.40	1
D	0.20	0

Positives and negatives alternate as we go down in score. $n_{+} = 2, n_{-} = 2$ . (Reminder from the shorthand box: $(+)$ = positive/ $y = 1$ , $(-)$ = negative/ $y = 0$ .)

Forecast: "Alternating" sounds like a coin flip → 0.5. Is it really?

Step 1. Sort descending and tag by true label: $A (+), B (-), C (+), D (-)$ . Why this step? Threshold sweep visits scores in this order, so we know exactly when TPR vs FPR steps up. The $(+)$ / $(-)$ tag tells us which rate moves at each crossing.

Step 2. Walk down, TPR up on a $(+)$ , FPR up on a $(-)$ :

Cross $A (+)$ → $(0, 0.5)$
Cross $B (-)$ → $(0.5, 0.5)$
Cross $C (+)$ → $(0.5, 1.0)$
Cross $D (-)$ → $(1.0, 1.0)$

Why this step? Each crossed example bumps exactly one rate; reading off its label tag tells us which axis moves.

Step 3. Trapezoid rule on the full point chain $(0, 0) \to (0, 0.5) \to (0.5, 0.5) \to (0.5, 1.0) \to (1.0, 1.0)$ . I'll take each segment in order and show its width and heights:

$(0, 0) \to (0, 0.5)$ : width $0$ → area $0$ (vertical rise, encloses nothing).
$(0, 0.5) \to (0.5, 0.5)$ : width $0.5$ , heights $0.5$ and $0.5$ → area $= 0.5 \cdot \frac{0.5 + 0.5}{2} = 0.25$ .
$(0.5, 0.5) \to (0.5, 1.0)$ : width $0$ → area $0$ (vertical rise).
$(0.5, 1.0) \to (1.0, 1.0)$ : width $0.5$ , heights $1.0$ and $1.0$ → area $= 0.5 \cdot \frac{1.0 + 1.0}{2} = 0.5$ .

Total $= 0 + 0.25 + 0 + 0.5 =$ 0.75. Why this step? Only the two horizontal segments have non-zero width, so only they contribute; the two vertical rises (where FPR does not change) add exactly 0, per the trapezoid rule.

Verify: Pairing — the positives are A=0.8, C=0.4; the negatives are B=0.6, D=0.2. Pairs: (A,B) $0.8 > 0.6$ →1; (A,D) $0.8 > 0.2$ →1; (C,B) $0.4 > 0.6$ →0; (C,D) $0.4 > 0.2$ →1. Sum $= 3$ , so $AUC = 3/4 = 0.75$ . ✓ Geometry and pairing agree.

Lesson: "Alternating labels" is not the same as "random scores." Three of the four pairs still rank the positive higher, so the model does carry signal (AUC $0.75 > 0.5$ ). To truly get $0.5$ you need genuine ties or fully interleaved scores — see Ex 4 for how ties push AUC downward.

Example 4 — Tied scores, the 0.5 rule (Cell C4)

Worked example A positive and a negative share a score

Ex	Score	Label
A	0.70	1
B	0.50	0
C	0.50	1
D	0.30	0

B and C tie at 0.50, opposite labels. $n_{+} = 2, n_{-} = 2$ .

Forecast: Ties should shave AUC toward 0.5. Above or below the no-tie value of 1.0?

Step 1. Pairing (cleanest for ties). Pairs (positive, negative):

(A, B): $0.70 > 0.50$ → $1$
(A, D): $0.70 > 0.30$ → $1$
(C, B): $0.50 = 0.50$ → tie → $0.5$
(C, D): $0.50 > 0.30$ → $1$

Why this step? Ties are awkward on the geometry staircase (they create a slanted segment), but the Mann–Whitney rule handles them cleanly: a tie is worth exactly half a win.

Step 2. Sum $= 1 + 1 + 0.5 + 1 = 3.5$ . Divide by $n_{+} n_{-} = 4$ : $AUC = \frac{3.5}{4} = 0.875$

Verify (geometry): the tie makes the curve go diagonally up-and-right for the B/C pair instead of a clean right-angle step. Point chain: $(0, 0) \to (0, 0.5)$ after A, then a diagonal $(0, 0.5) \to (0.5, 1.0)$ absorbing the B/C tie, then $(0.5, 1.0) \to (1.0, 1.0)$ after D. Trapezoid areas: $(0, 0.5) \to (0.5, 1.0)$ : width $0.5$ , heights $0.5$ and $1.0$ → $0.5 \cdot \frac{0.5 + 1.0}{2} = 0.375$ ; $(0.5, 1.0) \to (1.0, 1.0)$ : width $0.5$ , heights $1.0, 1.0$ → $0.5$ . Total $= 0.375 + 0.5 = 0.875$ . ✓ The tie cost us $0.125$ versus if C had cleanly beaten B ( $4/4 = 1.0$ ).

Example 5 — Degenerate: one class is empty (Cell C5)

Worked example All examples are positive

Ex	Score	Label
A	0.9	1
B	0.5	1
C	0.2	1

$n_{+} = 3, n_{-} = 0$ . What is FPR? What is AUC?

Forecast: Something breaks. What exactly?

Step 1. $FPR = \frac{F P}{n _{-}} = \frac{F P}{0}$ . Why this step? FPR needs "of all negatives, what fraction we flagged." With zero negatives, "fraction of nothing" is undefined — you cannot divide by 0.

Step 2. The pairing formula has $n_{+} n_{-} = 3 \times 0 = 0$ pairs. AUC $= \frac{0}{0}$ — undefined, not 0, not 1. Why this step? AUC compares positives against negatives; with no negatives there is literally nothing to compare against, so no number exists.

Verify / rule: AUC requires both classes present. If a test fold accidentally has one class only, ROC-AUC is meaningless — libraries throw an error or return nan. Sanity: the ROC curve lives in a square whose x-axis (FPR) can't even be defined, so there's no curve to measure. This is the boundary case you must guard against in code (check $n_{+} > 0$ and $n_{-} > 0$ ).

Example 6 — Both methods must agree (Cell C6)

Worked example Trapezoid vs Mann–Whitney on the parent note's 8 points

Reuse the parent note's dataset (one row per example; $(+)$ = positive/ $y = 1$ , $(-)$ = negative/ $y = 0$ ):

Ex	Score	Label
A	0.90	1
B	0.80	1
C	0.70	0
D	0.60	1
E	0.55	0
F	0.50	1
G	0.30	0
H	0.10	0

$n_{+} = 4, n_{-} = 4$ . We will compute AUC both ways and demand they match.

Forecast: They must land on the same value. Guess it before reading on.

Step 1 (pairing). Positives $(+)$ = {A:0.9, B:0.8, D:0.6, F:0.5}. Negatives $(-)$ = {C:0.7, E:0.55, G:0.3, H:0.1}. Count wins (positive $>$ negative, no ties here):

A(0.9): beats C, E, G, H → 4
B(0.8): beats C, E, G, H → 4
D(0.6): beats E(0.55), G, H, loses to C(0.7) → 3
F(0.5): beats G, H, loses to C, E → 2

Total wins $= 4 + 4 + 3 + 2 = 13$ , so $AUC = \frac{13}{16} = 0.8125$ . Why this step? Each positive's "wins" are just how many negatives it out-ranks; summing gives the numerator of the Mann–Whitney formula.

Step 2 (build the honest ROC table). Sweep $t$ down; each row is $T P, F P$ and the two rates:

$t$	TP	FP	TPR=TP/4	FPR=FP/4
0.90	1	0	0.25	0.00
0.80	2	0	0.50	0.00
0.70	2	1	0.50	0.25
0.60	3	1	0.75	0.25
0.55	3	2	0.75	0.50
0.50	4	2	1.00	0.50
0.30	4	3	1.00	0.75
0.10	4	4	1.00	1.00

Why this step? AUC via geometry needs the ordered $(FPR, TPR)$ points; this table is exactly that chain.

Step 3 (trapezoid rule — every segment, carefully). AUC only gains area where FPR changes. FPR takes the distinct values $0, 0.25, 0.5, 0.75, 1.0$ (each step width $0.25$ ). For each step I use the TPR at the start and end of that step:

FPR $0 \to 0.25$ : at FPR=0 the highest TPR reached is $0.5$ ; at FPR=0.25 the TPR is $0.75$ . Area $= 0.25 \cdot \frac{0.5 + 0.75}{2} = 0.25 \cdot 0.625 = 0.15625$ .
FPR $0.25 \to 0.5$ : TPR goes $0.75 \to 1.0$ . Area $= 0.25 \cdot \frac{0.75 + 1.0}{2} = 0.25 \cdot 0.875 = 0.21875$ .
FPR $0.5 \to 0.75$ : TPR stays $1.0 \to 1.0$ . Area $= 0.25 \cdot \frac{1.0 + 1.0}{2} = 0.25$ .
FPR $0.75 \to 1.0$ : TPR stays $1.0 \to 1.0$ . Area $= 0.25$ .

Total $= 0.15625 + 0.21875 + 0.25 + 0.25 =$ 0.8125. Why this step? We take the TPR at each FPR level to be the highest one reached there (the top of any vertical rise), then connect consecutive levels with trapezoids. The vertical rises themselves have width 0 and add nothing.

Verify: Geometry $= 0.8125$ and pairing $= 13/16 = 0.8125$ — identical. ✓ The two stories always agree when the arithmetic is done cleanly; earlier confusion came from mis-reading which TPR sits at each FPR level, which this careful table fixes.

Example 7 — Severe imbalance (Cell C7)

Worked example 1 positive, 9 negatives — ROC looks perfect, alarms flood you

A fraud model scores 10 transactions. Only 1 is fraud (label 1). Scores:

Ex	Score	Label
Fraud	0.85	1
N1..N9	0.80, 0.70, 0.60, 0.50, 0.40, 0.30, 0.20, 0.10, 0.05	0

The fraud (0.85) outscores all 9 legit transactions. $n_{+} = 1, n_{-} = 9$ .

Forecast: ROC-AUC will look excellent. Is that the whole story?

Step 1. Pairing: fraud (0.85) beats every negative → 9 wins out of $n_{+} n_{-} = 1 \times 9 = 9$ . $AUC = 9/9 = 1.0$ . Why this step? With one positive, AUC just asks "did the single fraud outrank each legit transaction?" — yes, all of them, so the ranking is perfect.

Step 2. Now inspect the operating point you'd actually deploy at, threshold $t = 0.5$ : predict fraud for scores $\geq 0.5$ = {Fraud, N1, N2, N3, N4} (scores 0.85, 0.80, 0.70, 0.60, 0.50). That's $T P = 1$ , $F P = 4$ .

$FPR = 4/9 \approx 0.444$ — but in raw counts, 4 false alarms for every 1 real fraud caught.
Precision $= \frac{T P}{T P + F P} = \frac{1}{1 + 4} = 0.2$ — only 20% of your flags are real fraud.

Why this step? AUC hid the pain because FPR normalises by the huge negative count. Precision (see Precision-Recall Curve and F1-Score) does not, so it exposes how few of your alarms are true.

Verify: ROC-AUC $= 1.0$ (perfect ranking) yet precision-at-threshold $= 0.2$ . Both numbers are true and do not contradict — they answer different questions ("ranking quality" vs "purity of alarms"). On imbalanced data, report PR-AUC alongside ROC-AUC. ✓

Example 8 — Real-world cost twist (Cell C8)

Worked example Choosing a threshold when a miss costs 10× a false alarm

Same 8-point dataset as Ex 6. A missed positive ( $F N$ ) costs $10; a false alarm ( $F P$ ) costs $1. Which threshold minimises expected cost?

Forecast: Because misses are 10× worse, we should lean toward a low threshold (catch more). Guess which $t$ wins.

Step 1. Cost at threshold $t$ : $Cost (t) = 10 \cdot F N (t) + 1 \cdot F P (t)$ , with $F N = n_{+} - T P = 4 - T P$ . Why this step? This is Cost-Sensitive Learning — we weight each error type by its dollar cost instead of treating them equally, then minimise total dollars.

Step 2. Tabulate from Ex 6's counts:

$t$	TP	FP	FN=4−TP	Cost = 10·FN + FP
0.90	1	0	3	30
0.80	2	0	2	20
0.60	3	1	1	11
0.50	4	2	0	2
0.30	4	3	0	3
0.10	4	4	0	4

Why this step? We just read $T P, F P$ off the Ex 6 sweep and plug into the cost formula; the smallest number is our choice.

Step 3. Minimum cost is at $t = 0.50$ : $2, catching all 4 positives with only 2 false alarms. Why this step? Below $0.50$ we gain no more true positives (all 4 already caught) but keep adding false alarms; above it we start missing costly positives.

Verify: As forecast, the cost-optimal threshold is low ( $0.50$ ), not the "balanced" $0.70$ . Units check: 0 misses × $10 + 2 alarms × $1 = $2. ✓ Dropping to $t = 0.30$ adds a needless false alarm ( $+\$ 1 \to $3 $), so$ 0.50$ wins.

Example 9 — Exam trap: high accuracy, useless AUC (Cell C9)

Worked example The "always healthy" classifier

100 patients: 95 healthy (label 0), 5 sick (label 1). Model always outputs score 0.10 for everyone (it learned nothing).

Forecast: Accuracy will look great. What's the AUC?

Step 1. Predict "healthy" for all (any threshold $> 0.10$ ). Then $T P = 0, F P = 0, T N = 95, F N = 5$ . $Accuracy = \frac{T P + T N}{100} = \frac{0 + 95}{100} = 0.95 = 95%$ Why this step? Accuracy rewards getting the majority class right, and 95% of patients are healthy, so doing nothing scores 95%.

Step 2. AUC by pairing: every sick patient and every healthy patient share the same score 0.10 → every one of the $5 \times 95 = 475$ pairs is a tie worth $0.5$ . $AUC = \frac{475 \times 0.5}{475} = 0.5$ Why this step? Identical scores mean the model cannot rank anyone above anyone — pure coin-flip separation, which is exactly what AUC $= 0.5$ encodes.

Verify: Accuracy $= 0.95$ but AUC $= 0.5$ . The model catches zero of the 5 sick patients — clinically useless — yet "95% accurate." This is exactly the parent note's Mistake 1. AUC exposed the fraud that accuracy hid. ✓ (For multi-class versions of this trap, see Multi-Class Classification.)

Recall Quick self-test

Perfect ranking gives AUC = ? ::: 1.0 Backwards ranking gives AUC = ? ::: 0.0 (flip predictions to get 1.0) A positive tied with a negative contributes how much to the pairing sum? ::: 0.5 Why is AUC undefined when $n_{-} = 0$ ? ::: FPR divides by $n_{-} = 0$ , and the pairing formula divides by $n_{+} n_{-} = 0$ . On 95%-healthy data, an "always healthy" model has accuracy ≈ ? and AUC = ? ::: accuracy 0.95, AUC 0.5 When ROC-AUC looks perfect on imbalanced data, which metric reveals the false-alarm flood? ::: Precision (PR-AUC)