ROC curve and AUC

The Core Concept

A Receiver Operating Characteristic (ROC) curve plots the performance of a binary classifier as we sweep its decision threshold from 0 to 1. On the y-axis: True Positive Rate (TPR), also called sensitivity or recall. On the x-axis: False Positive Rate (FPR).

WHY these axes?

TPR answers: "Of all actual positives, what fraction did we catch?" (We WANT this high)
FPR answers: "Of all actual negatives, what fraction did we incorrectly flag?" (We WANT this low)

As you lower the threshold (predict "positive" more often), BOTH rates increase—but at different speeds depending on how good your model is.

Definition Mathematical Definitions

Given a confusion matrix at threshold $t$ : $\text{TPR}(t) = \frac{TP}{TP + FN} = \frac{\text{True Positives}}{\text{All Actual Positives}}$ $\text{FPR}(t) = \frac{FP}{FP + TN} = \frac{\text{False Positives}}{\text{All Actual Negatives}}$ The ROC curve is the set of points $\{(\text{FPR}(t), \text{TPR}(t)) : t \in [0,1]\}$ .

The Area Under the Curve (AUC) is: $\text{AUC} = \int_0^1 \text{TPR}(\text{FPR}^{-1}(x)) \, dx$ In practice, computed as the trapezoidal area under the empirical ROC curve.

WHY this integral? The AUC measures the total "lift" your classifier provides over random guessing across all operating points. It equals the probability that a randomly chosen positive example ranks higher than a randomly chosen negative example.

Deriving the ROC Curve from First Principles

Setup: You have a trained binary classifier that outputs scores $s_i \in [0,1]$ for examples $i=1,\ldots,n$ . True labels are $y_i \in \{0,1\}$ . Let $n_+ = \sum_i \mathbb{1}[y_i=1]$ and $n_- = \sum_i \mathbb{1}[y_i=0]$ .

Step 1: Pick a threshold $t$ . Predict $\hat{y}_i = 1$ if $s_i \geq t$ , else $\hat{y}_i = 0$ .

Step 2: Count the confusion matrix entries. $TP(t) = \sum_{i: y_i=1} \mathbb{1}[s_i \geq t]$ $FP(t) = \sum_{i: y_i=0} \mathbb{1}[s_i \geq t]$ $FN(t) = \sum_{i: y_i=1} \mathbb{1}[s_i < t] = n_+ - TP(t)$ $TN(t) = \sum_{i: y_i=0} \mathbb{1}[s_i < t] = n_- - FP(t)$

WHY this counting? Each predicted positive with true label 1 is aTP; each predicted positive with true label 0 is an FP. The threshold determines which predictions are positive.

Step 3: Compute rates. $\text{TPR}(t) = \frac{TP(t)}{n_+}, \quad \text{FPR}(t) = \frac{FP(t)}{n_-}$

Step 4: Vary $t$ from 1 down to 0.

At $t=1$ : No predictions are positive → $TP=0, FP=0$ → point $(0,0)$
At $t=0$ : All predictions are positive → $TP=n_+, FP=n_-$ → point $(1,1)$
Each unique score in your dataset creates a new point on the ROC curve.

WHY does lowering $t$ move us up and right? As $t$ decreases,\mathbb{1}[s_i \geq t]$ becomes1 for more examples. BothTP and FP counts increase (never decrease). So TPR and FPR both increase monotonically.

Formula AUC as Probabilistic Interpretation

$\text{AUC} = P(s_i > s_j \mid y_i=1, y_j=0)$ Derivation: Consider all pairs $(i,j)$ where $y_i=1, y_j=0$ . The AUC equals the fraction of such pairs where the positive example scores higher.

WHY? Each point on the ROC curve corresponds to a threshold. The area under the curve aggregates the "separation quality" over all thresholds. If positives always score higher than negatives, every pair is correctly ordered → AUC = 1. If scores are random, half the pairs are correctly ordered → AUC = 0.5.

Computational formula (Mann-Whitney U): $\text{AUC} = \frac{1}{n_+ n_-} \sum_{i:y_i=1} \sum_{j:y_j=0} \mathbb{1}[s_i > s_j] +0.5 \cdot \mathbb{1}[s_i = s_j]$

Interpreting AUC Values

AUC Range	Interpretation
0.9 - 1.0	Excellent discrimination
0.8 - 0.9	Good discrimination
0.7 - 0.8	Acceptable
0.5 - 0.7	Poor (better than random, but barely useful)
0.5	Random classifier (diagonal line)
< 0.5	Worse than random (you can flip predictions to get >0.5)

WHY is 0.5 the baseline? A random classifier assigns scores uniformly at random. On average, it ranks a positive higher than a negative 50% of the time.

Worked example Worked Example 1: Computing ROC by Hand

Scenario: 8 examples with scores and true labels:

Example	Score	True Label
A	0.9	1
B	0.8	1
C	0.7	0
D	0.6	1
E	0.55	0
F	0.5	1
G	0.3	0
H	0.1	0

$n_+ = 4, n_- = 4$

Step 1: Sort by score descending (already done).

Step 2: For each unique threshold between scores, count TP and FP: | Threshold $t$ | Positives (score ≥ t) | TP | FP | TPR = TP/4 | FPR = FP/4 | |------------|----------------------|----|--------| | 1.0 | none | 0 | 0 | 0.00 | 0.00 | | 0.9 | A | 1 | 0.25 | 0.00 | | 0.8 | A,B | 2 | 0.50 | 0.00 | | 0.7 | A,B,C | 2 | 1 | 0.50 | 0.25 | | 0.6 | A,B,C,D | 3 | 1 | 0.75 | 0.25 | | 0.55 | A,B,C,D,E | 3 | 2 | 0.75 | 0.50 | | 0.5 | A,B,C,D,E,F | 4 | 2 | 1.00 | 0.50 | | 0.3 | all but H | 4 | 3 | 1.00 | 0.75 | | 0.0 | all | 4 | 4 | 1.00 | 1.00 |

WHY this step? Example C (score 0.7, label 0) is our first false positive. Before that, we catch two true positives (A, B) with zero false alarms.

Step 3: Plot these (FPR, TPR) points and connect them. The ROC curve goes: $(0,0) \to (0, 0.25) \to (0, 0.5) \to (0.25, 0.5) \to (0.25, 0.75) \to (0.5, 0.75) \to (0.5, 1.0) \to (0.75, 1.0) \to (1.0, 1.0)$

Step 4: Compute AUC via trapezoids: $\text{AUC} = \sum_i \frac{1}{2}(\text{FPR}_{i+1} - \text{FPR}_i)(\text{TPR}_i + \text{TPR}_{i+1})$ $= \frac{1}{2}(0.25)(0 + 0.5) + \frac{1}{2}(0.25)(0.5 + 0.75) + \frac{1}{2}(0.25)(0.75 + 1.0) + \frac{1}{2}(0.25)(1.0 + 1.0)$ $= 0.0625 + 0.15625 + 0.21875 + 0.25 = 0.6875$

Interpretation: AUC = 0.69 → acceptable but not great. The model ranks positives higher than negatives most of the time, but there's substantial overlap.

Worked example Worked Example 2: Comparing Two Models

Model A: Predicts cancer based on tumor size. AUC = 0.78 Model B: Predicts cancer based on tumor size + patient age + genetic markers. AUC = 0.91

WHY is AUC comparison valid here? Both models predict the same outcome on the same test set. AUC is threshold-agnostic, so we're not biasing the comparison by picking a favorable threshold for one model.

Decision: Model B is clearly better at separating cancer from non-cancer across all operating points. Even if you later need a high-precision threshold (low FPR) or high-recall threshold (high TPR), Model B will outperform Model A.

WHY this matters: In medical screening, you might deploy at high sensitivity (catch all cancers, tolerate false alarms) but use AUC during development to pick the best model architecture.

Connections to Other Metrics

The ROC curve unifies several concepts:

Precision-Recall Curve: Alternative visualization for imbalanced data (Precision vs. Recall instead of TPR vs. FPR).
Confusion Matrix: ROC curve is built from confusion matrices at many thresholds.
Sensitivity and Specificity: TPR = Sensitivity; (1 - FPR) = Specificity. ROC plots sensitivity vs. (1 - specificity).
F1-Score: Depends on a fixed threshold. ROC explores all thresholds. Use F1 at your deployed threshold, AUC to pick the model.
Cost-Sensitive Learning: If false positives and false negatives have different costs, find the ROC point that minimizes expected cost.
Multi-Class Classification: Extend to one-vs-rest ROC curves per class, or use macro-average AUC (average AUC across classes).

Recall Explain to a 12-Year-Old

Imagine you're a teacher grading essays as "good" or "bad" using a scoring robot. The robot gives each essay a score from 0 to 100. You need to pick a cutoff: "above 70 is good, below is bad."

But what if you're not sure whether70 is the right cutoff? Maybe60 catches more good essays (you don't want to miss any!), but it also accidentally calls some bad essays good.

The ROC curve is like trying EVERY possible cutoff—60, 65, 70, 75, 80—and seeing what happens. The y-axis shows "how many good essays did I catch?" (higher is better). The x-axis shows "how many bad essays did I accidentally call good?" (lower is better).

A perfect robot would catch all good essays without any mistakes—its curve would go straight up the left side, then across the top. A robot that just guesses randomly would have a diagonal line (catch 50% of good ones, but also call 50% of bad ones good).

AUC is the area under that curve. It's one number that tells you: "overall, across all cutoffs, how good is this robot at telling good essays from bad ones?" AUC = 1.0 is perfect, 0.5 is random guessing.

When to Use ROC vs. PR Curve

Scenario	Prefer ROC-AUC	Prefer PR-AUC
Balanced classes (≈50/50)	✓	Either
Imbalanced (rare positives)	✗ (can be misleading)	✓
Care equally about FP and FN	✓
Care MORE about positive class		✓
Need threshold-agnostic comparison	✓	✓

WHY? Precision = TP/(TP+FP) explicitly includes FP in the numerator's context. When positives are rare, even a low FPR can mean many absolute false positives, tanking precision. PR curves expose this; ROC curves can hide it.

#flashcards/ai-ml

What does the ROC curve plot on its axes? :: Y-axis: True Positive Rate (TPR), X-axis: False Positive Rate (FPR). It shows the tradeoff between sensitivity and false alarms across all thresholds.

What is AUC and what does it measure?

Area Under the ROC Curve. It measures the probability that a randomly chosen positive example ranks higher than a randomly chosen negative example. Equivalently, it's the classifier's separation ability aggregated over all thresholds.

Why is AUC = 0.5 the baseline for a useless classifier?

A random classifier ranks a positive above a negative exactly50% of the time (no separation ability). Its ROC curve is the diagonal line from (0,0) to (1,1), with area = 0.5.

How do you interpret AUC = 0.92?

Excellent discrimination. The model ranks 92% of (positive, negative) pairs correctly. It separates the two classes well across all operating points.

What happens to TPR and FPR as you lower the decision threshold?

BOTH increase. Lowering the threshold means predicting "positive" more often, so you catch more true positives (TPR up) but also flag more false positives (FPR up).

Why can AUC be misleading on highly imbalanced datasets?

FPR = FP/n₋. When n₋ is huge, even many absolute false positives yield a low FPR, making the ROC curve look great. But in practice, you're drowning in false alarms. Precision-Recall AUC exposes this better.

What is the probabilistic interpretation of AUC? :: AUC = P(score of random positive > score of random negative). It's the fraction of all (pos, neg) pairs where the positive scores higher.

If a model has 90% accuracy but AUC = 0.55, what's going on?

The model likely predicts the majority class almost always (high accuracy from class imbalance) but has near-zero ability to separate classes (AUC ≈ random). It's useless for identifying the minority class.

What's the difference between ROC-AUC and PR-AUC?

ROC-AUC uses TPR vs. FPR (both normalized by their class). PR-AUC uses Precision vs. Recall (Precision = TP/(TP+FP), sensitive to absolute FP count). PR-AUC is better for imbalanced data.

When does the ROC curve reach the point (0, 0)?

When threshold = 1.0 (predict nothing as positive). TP = 0, FP = 0, so TPR = 0, FPR = 0.

When does the ROC curve reach the point (1, 1)? :: When threshold = 0 (predict everything as positive). TP = n₊, FP = n₋, so TPR = 1, FPR = 1.

Why is ROC-AUC threshold-agnostic?

It aggregates performance over ALL possible thresholds by plotting (FPR, TPR) for every threshold and measuring the area under the resulting curve. You're not committing to a single threshold.

What does "the ROC curve hugs the top-left corner" mean?

The model achieves high TPR (y-axis) with low FPR (x-axis), meaning it catches most positives while triggering few false alarms. This is ideal classifier behavior.

How is AUC related to the Mann-Whitney U statistic?

They're equivalent. AUC = (U / (n₊ × n₋)) where U counts pairs where positive > negative. It's the same as the rank-sum interpretation.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

ROC curve aur AUC ko samajhne ke liye ek simple scenario socho: tumhare pas ek medical test hai jo cancer detect karta hai, aur wo har patient ko ek score deta hai 0 se 1 ke bech. Ab tum decide karna hai kitne score pe tum bologe "haan, cancer hai"—kya 0.5 pe? 0.7 pe? 0.9 pe? Har threshold pe kuch patients sahi pakde jayenge (true positives) aur kuch galat alarm bajegi (false positives).

ROC curve ye dikhata hai ki SARE possible thresholds pe tumhara model kaisa perform karta hai. Y-axis pe hai "kitne actual cancer patients pakde" (TPR yani sensitivity), aur X-axis pe hai "kitne healthy logon ko galti se cancer bola" (FPR). Jab tum threshold neeche karte ho (zyada lenient decision), dono rates badhte hain—zyada patients catch hote hain, lekin zyada false alarms bhi. Ek acha model left-top corner ko touch karega (sare actual positives pakde, zero false alarms)—wo perfect model hoga. Random guessing wala model ek diagonal line dega (50-50 chance), aur uska AUC sirf 0.5 hoga.

AUC (Area Under the Curve) uss pore curve ke neeche ka area hai—ek single number jo bata hai: "overall, model kitna acha hai positives ko negatives se alag karne mein, chahe tum konsa bhi threshold chuno." Agar AUC = 0.9 hai, matlab90% cases mein model ne ek random positive ko ek random negative se zyada score diya—yani excellent discrimination. Medical tests mein, fraud detection mein, spam filtering mein—jahan bhi tumhe probabilities milte hain aur tum threshold decide karna hai—ROC curve tumhe bata hai ki konsa model best hai, bina kisi ek threshold pe commit kiye.

Imbalanced data (jahan positive cases bahut kam hain) mein, sirf ROC-AUC dekhna misleading ho sakta hai kyunki FPR normalize ho jata hai bade negative class se. Wahan Precision-Recall curve b

The Core Concept

Deriving the ROC Curve from First Principles

Interpreting AUC Values

Connections to Other Metrics

When to Use ROC vs. PR Curve

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Model Evaluation & Selection

Connections