ROC curve and AUC: Visual walkthrough — AI-ML deep dive

Step 0 — The raw material: scores and true answers

Before any curve exists, all we have is a list. For each thing our model looked at (an email, a tumour, a transaction) it printed a score — a number between 0 and 1 saying "how sure am I this is a POSITIVE?". Separately, reality has a true label: this thing really was a positive (label $1$ ) or a negative (label $0$ ).

We use the same 8 examples as Worked Example 1 of the parent. Look at the figure: each example is a dot on a number line, placed at its score. Coral dots are true positives, slate dots are true negatives.

WHAT: we laid every example on a horizontal score-axis and coloured it by its true label. WHY: the entire ROC (Receiver Operating Characteristic) story is about how well the colours separate along this line. If all coral sat to the right of all slate, the model would be perfect. Notice they overlap — that overlap is exactly what AUC will measure. PICTURE: coral dot $C$ ? No — $C$ is slate (a negative) sitting at score $0.7$ , higher than the coral dot $D$ at $0.6$ . That single crossing is a mistake waiting to hurt our score.

Step 1 — The threshold: a sliding wall

We cannot leave scores as fuzzy confidences forever; at some point we must decide. We pick a number $t$ , the threshold, and rule:

$\overset{y}{^}_{i} = {10 if s_{i} \geq t (call it positive) if s_{i} < t (call it negative)$

$t$ ::: the threshold, the cut-off line we slide along the score-axis.
$\overset{y}{^}_{i}$ ::: our decision (the hat means "predicted"), as opposed to the truth $y_{i}$ .
$s_{i} \geq t$ ::: "score is at or past the wall" → we predict positive.

WHAT: we dropped a vertical wall at $t = 0.65$ ; everything to its right is predicted positive. WHY: this turns fuzzy scores into hard yes/no calls so we can count how many we got right. PICTURE: to the right of the wall sit $A, B$ (coral, correct!) and $C$ (slate, a false alarm). The wall's position is the only knob we will turn.

Step 2 — Four buckets: the confusion counts

Once the wall is placed, every example falls into exactly one of four buckets, decided by two yes/no questions: did we call it positive? and was it really positive?

$T P (t) = \sum_{i : y_{i} = 1} 1 [s_{i} \geq t], F P (t) = \sum_{i : y_{i} = 0} 1 [s_{i} \geq t]$

Reading this term by term:

$\sum_{i : y_{i} = 1}$ ::: "walk through only the truly positive examples".
$1 [s_{i} \geq t]$ ::: the indicator — it equals $1$ when the bracket is true, $0$ otherwise. So the sum just counts how many truly-positive examples landed right of the wall.
$F P$ : same walk, but over truly negative examples — counting the false alarms.

This is the Confusion Matrix evaluated at one threshold.

WHAT: we split the line into a $2 \times 2$ grid of coloured buckets at $t = 0.65$ . WHY: every performance number we ever compute is built only from these four counts — nothing else exists. PICTURE: at this wall, $T P = 2$ (A, B), $F P = 1$ (C), $F N = 2$ (D, F), $T N = 3$ (E, G, H). Two catches, one false alarm.

Step 3 — Two rates: the axes of the curve

Raw counts depend on how big your dataset is. To make them comparable we turn each into a rate by dividing by its own class total.

$y-axis TPR (t) = \frac{T P ( t )}{n _{+}}, x-axis FPR (t) = \frac{F P ( t )}{n _{-}}$

$TPR$ ::: True Positive Rate = of all real positives, the fraction we caught. Its one alias is sensitivity (also called recall). We want this high.
$n_{+}$ in the denominator ::: divides by all actual positives, so TPR is "out of the catchable ones".
$FPR$ ::: False Positive Rate = of all real negatives, the fraction we wrongly flagged. We want this low.
$n_{-}$ in the denominator ::: divides by all actual negatives, so a huge negative class shrinks each false alarm's impact (remember this — it becomes the imbalance trap).

WHAT: we mapped the counts at $t = 0.65$ to a single dot at $(FPR, TPR) = (0.25, 0.50)$ . WHY: one wall position → one dot in a square. Slide the wall, get more dots — those dots are the ROC curve. PICTURE: the dot lives in the unit square. Its height $0.5$ says "caught half the positives"; its horizontal $0.25$ says "falsely flagged a quarter of negatives".

Step 4 — Sweep the wall: the curve is born

Now we move the wall from far right ( $t = 1$ , nobody predicted positive) to far left ( $t = 0$ , everybody predicted positive), pausing at every score. Each pause gives one dot; we connect them.

WHY does lowering $t$ only ever move us up-and-right (never back)? As $t$ drops, more examples cross to the "predicted positive" side. Each newly-crossed example is either a coral (bumps TPR up) or a slate (bumps FPR right). Nothing ever leaves the positive side, so both rates are monotone — the curve never dips or backtracks. A coral crossing draws a vertical step; a slate crossing draws a horizontal step.

WHAT: we drew all nine dots from the parent's table and joined them into the staircase ROC curve. WHY: every vertical rise is a true positive earned "for free" (no new false alarm); every horizontal jog is the price we pay in false alarms. PICTURE: the first two steps go straight up (we caught A, B with zero false alarms — pure profit). Then $C$ crosses: a step right. That rightward jog is exactly the slate dot that sat too high in Step 0.

Step 5 — Measure the area: AUC by trapezoids

The whole curve squeezed into one number is the Area Under the Curve. Since the curve is a chain of straight segments, the area under it is a chain of trapezoids.

$AUC = \sum_{i} \frac{1}{2} width of strip (FPR_{i + 1} - FPR_{i}) sum of two heights (TPR_{i} + TPR_{i + 1})$

the width term ::: how far right this segment travels (zero for vertical steps → they add no area).
the sum of heights term ::: the two parallel sides of the trapezoid; the outer $\frac{1}{2}$ turns this sum into their average.
Only horizontal moves (false-positive steps) contribute width, and the height at that moment is how many positives you had already banked.

Careful with the count: there are $n_{-} = 4$ negatives, so there are exactly four rightward segments (one per negative crossing). Their left FPRs are $0, 0.25, 0.5, 0.75$ and each has width $0.25$ . Plugging in the parent's points, keeping all four:

$AUC = \frac{1}{2} (0.25) (0.5 + 0.5) + \frac{1}{2} (0.25) (0.5 + 0.75) + \frac{1}{2} (0.25) (0.75 + 1.0) + \frac{1}{2} (0.25) (1.0 + 1.0)$ $= 0.0625 + 0.15625 + 0.21875 + 0.25 = 0.6875$

(the earlier steps read off the staircase: heights $0.5, 0.5, 0.75, 1.0$ on the left edge of each rightward segment, rising to $0.5, 0.75, 1.0, 1.0$ on the right edge.)

WHAT: we shaded the four trapezoid strips under the staircase and summed them to $0.6875$ . WHY: the total shaded fraction of the unit square is the AUC — a bigger area means positives generally out-rank negatives. PICTURE: notice tall strips appear only after we have banked positives — the area rewards catching positives before you start collecting false alarms.

Step 6 — Why the area IS the probability (the visual bridge)

Here is the beautiful twist. That geometric area equals a betting probability:

$AUC = P (s_{i} > s_{j} y_{i} = 1, y_{j} = 0)$

In words: pick one real positive and one real negative at random; AUC is the chance the positive got the higher score. Why must the area equal this count? Watch one vertical rise.

Every time the wall crosses one negative, it steps right by exactly $\frac{1}{n _{-}}$ . The height of the curve at that moment is $TPR = \frac{( positives already banked )}{n _{+}}$ . The thin strip's area is therefore

$one negative \frac{1}{n _{-}} \times their fraction \frac{# { positives above this negative }}{n _{+}} = \frac{# { positives that out-score this negative }}{n _{+} n _{-}} .$

$\frac{1}{n _{-}}$ ::: the width, "we just processed one negative $j$ ".
the fraction ::: exactly how many positives already crossed — i.e. how many positives beat this negative.

Sum the strips over all negatives and you have added up, for every negative, the positives that beat it — divided by $n_{+} n_{-}$ . That is precisely the count of correctly-ordered pairs over the total number of pairs. So area = fraction of winning pairs = probability. The Confusion Matrix counting and the geometry are the same act viewed two ways.

Mann–Whitney U form (with the tie term from Step 4). Let $P = {i : y_{i} = 1}$ be the set of positive indices and $N = {j : y_{j} = 0}$ be the set of negative indices; the outer sum runs over every positive index $i \in P$ and the inner sum over every negative index $j \in N$ , so together they visit all $n_{+} n_{-}$ positive–negative pairs exactly once:

$AUC = \frac{1}{n _{+} n _{-}} \sum_{i \in P} \sum_{j \in N} (1 [s_{i} > s_{j}] + \frac{1}{2} 1 [s_{i} = s_{j}])$

$\sum_{i \in P}$ ::: walk over each positive example $i$ .
$\sum_{j \in N}$ ::: for that positive, walk over each negative example $j$ .
$1 [s_{i} > s_{j}]$ ::: award $1$ if the positive out-scored the negative (a "win").
$\frac{1}{2} 1 [s_{i} = s_{j}]$ ::: award $\frac{1}{2}$ for a tie — the shared credit from Step 4.
$\frac{1}{n _{+} n _{-}}$ ::: divide by the total number of pairs to turn the count into a fraction.

For our 8 examples there are $n_{+} n_{-} = 4 \times 4 = 16$ pairs; counting correctly-ordered pairs gives $11$ → $11/16 = 0.6875$ . Same number as the trapezoid area — geometry and probability agree.

WHAT: a grid of all 16 positive-vs-negative pairs; a cell is mint when the positive out-scored the negative, coral when it lost. The dashed strip highlights that one negative's column = one vertical rise of the staircase. WHY: it shows AUC is literally "fraction of the grid coloured mint" — the same quantity the shaded area measured, negative-column by negative-column. PICTURE: the coral cells are exactly the pairs where a negative (like $C$ at $0.7$ ) beat a positive (like $D$ at $0.6$ ) — the crossings you spotted way back in Step 0.

Step 7 — The degenerate cases (never get surprised)

Each corner case is drawn in the figure below — one panel per case, so you can match the shape to the story.

Definition Every corner case, drawn

Perfect model (top-left panel) — all coral score above all slate. Curve hugs the top-left corner, $AUC = 1$ . Every pair mint.
Random model (top-right panel) — scores unrelated to labels. Curve wanders the diagonal, $AUC = 0.5$ . Half the grid mint.
Inverted model (bottom-left panel) — coral consistently score below slate. Curve sags below the diagonal, $AUC < 0.5$ . Fix: flip the predictions and get $1 - AUC$ .
All-ties (bottom-right panel) — model outputs one constant score for everyone. Every pair is a tie → each worth $\frac{1}{2}$ → $AUC = 0.5$ . The curve is the straight diagonal (a single all-diagonal step, exactly as Step 4 predicts).
One class empty ( $n_{+} = 0$ or $n_{-} = 0$ ) — a rate divides by zero; AUC is undefined. You cannot rank against a class that isn't there, so there is no curve to draw.

WHAT: four ROC panels — perfect (corner), random (diagonal), inverted (below), all-ties (diagonal). WHY: so you recognise instantly what any curve's shape is telling you before reading a number. PICTURE: the diagonal is the "know-nothing" line; distance above it is skill, distance below it is skill pointing the wrong way.

The one-picture summary

This single figure chains the whole journey: dots on a line → sliding wall → four buckets → two rates → staircase curve → shaded area → paired-comparison grid. Read left to right and you have re-derived ROC and AUC from nothing but counting.

Recall Feynman retelling — say it back in plain words

I start with a bunch of dots on a line, coloured by whether they're really positive or negative, placed by how confident the model is. Then I imagine a wall I can slide left and right; everything right of the wall I declare positive. At each wall spot I count four things: how many positives I caught (TP), how many negatives I falsely accused (FP), and the two I missed. I turn those into two fractions — the height "fraction of positives caught" (TPR, a.k.a. sensitivity) and the width "fraction of negatives falsely flagged" (FPR, which is one-minus-specificity). Each wall position is one dot in a square. Sliding the wall from far-right to far-left traces a staircase that only ever climbs up (good) or steps right (costly) — and if a positive and negative tie, it slides diagonally, splitting the credit. The area under that staircase is the AUC. Here's the magic: each rightward step's little strip counts exactly how many positives already beat that one negative, so summing all strips counts all winning pairs — the area literally is the chance a random positive out-scores a random negative. Perfect separation = area 1; coin-flip = area 0.5; upside-down model = below 0.5 (just flip it). And if my negatives massively outnumber my positives, the area flatters me — so I'd better also look at precision–recall.

Recall Quick self-check

Why does a vertical step in the ROC curve add zero area? ::: Because its width $(FPR_{i + 1} - FPR_{i}) = 0$ ; only rightward (false-positive) moves have width. What does AUC $= 0.5$ mean geometrically and probabilistically? ::: Geometrically the curve lies on the diagonal; probabilistically a random positive out-ranks a random negative exactly half the time. Is specificity just another name for TPR? ::: No — specificity is the True Negative Rate $= T N / n_{-} = 1 - FPR$ ; only sensitivity is another name for TPR. How do you draw the staircase step when a positive and a negative share a score? ::: As a single diagonal to $(FPR + \frac{1}{n _{-}}, TPR + \frac{1}{n _{+}})$ , giving each tied pair half credit.

See also: Confusion Matrix · Sensitivity and Specificity · F1-Score · Precision-Recall Curve · Multi-Class Classification · Cost-Sensitive Learning