2.6.9 · D5Model Evaluation & Selection
Question bank — ROC curve and AUC
Before we start, three words the whole page leans on, in plain language:
See Confusion Matrix, Sensitivity and Specificity if any of those feels shaky.
True or false — justify
A model with 95% accuracy always has AUC near 0.95.
False — on imbalanced data a "predict the majority class always" model gets high accuracy but AUC ; accuracy is threshold-and-balance dependent, AUC is neither.
AUC = 0.5 means the model is broken and outputs nothing useful.
Not broken, just non-discriminating — it ranks a random positive above a random negative exactly half the time, the same as coin-flip scoring.
An AUC of 0.3 means the model is useless.
False — it means the model is systematically wrong; flipping every prediction turns 0.3 into 0.7, so the information is real, just inverted.
The ROC curve must always pass through and .
True — at above every score nothing is flagged (0 TP, 0 FP), and at below every score everything is flagged (all TP, all FP), forcing those two endpoints.
Lowering the threshold can decrease the TPR.
False — lowering only adds examples to the "predicted positive" set, so TP can only rise or stay; TPR is monotonically non-decreasing.
A perfect classifier's ROC curve is the diagonal line.
False — the diagonal is the random classifier; a perfect one hugs the top-left corner, going with AUC = 1.
If Model A has higher AUC than Model B, Model A is better at every threshold.
False — a higher AUC means better on average; two curves can cross, so B may still win in a specific FPR region you care about.
AUC equals the probability a random positive scores higher than a random negative.
True — this is the Mann–Whitney interpretation; it's why AUC measures ranking quality, not calibrated probabilities.
Changing the class balance of the test set changes the ROC curve.
False — TPR uses only positives and FPR uses only negatives, each normalized within its own class, so the curve is invariant to the positive:negative ratio.
Two models with identical AUC are interchangeable.
False — same area can come from very different curve shapes; one may excel at low FPR (screening) and the other at high TPR, so the use case decides.
Spot the error
"AUC = 0.88, so 88% of predictions are correct."
AUC is not an accuracy percentage; it's a ranking probability. Correct-prediction fraction depends on a chosen threshold, which AUC never fixes.
"My fraud detector has ROC-AUC = 0.97, so it's production-ready."
With 0.1% fraud, a tiny FPR rate is still a flood of false alarms in absolute count; check Precision-Recall Curve / PR-AUC before trusting it — see Cost-Sensitive Learning.
"I'll pick the threshold that maximizes AUC."
AUC is computed over all thresholds at once — it has no single threshold to maximize; you pick a threshold from a point on the curve, guided by cost, not by AUC.
"FPR + TPR = 1 always, because they're complementary rates."
They are not complements; FPR relates to negatives, TPR to positives. The true complement of FPR is specificity (), see Sensitivity and Specificity.
"To improve AUC I lowered the threshold."
The threshold does not change the curve or its area — it only chooses where you sit on the fixed curve; improving AUC needs a better model, not a new cutoff.
"ROC-AUC and F1 measure the same thing."
F1-Score is at one threshold and combines precision with recall; ROC-AUC is threshold-free and ignores precision entirely, so they can disagree sharply on imbalanced data.
"For 3 classes I'll just read one ROC curve."
A single ROC is binary; multi-class needs one-vs-rest or one-vs-one curves and an averaging scheme — see Multi-Class Classification.
Why questions
Why is 0.5 the AUC baseline rather than 0?
Random scoring still orders half of all positive-negative pairs correctly by luck, so 0.5 (not 0) is the "no skill" floor; 0 would require perfectly inverted ranking.
Why does AUC not require choosing a threshold, while accuracy does?
AUC integrates over every threshold via the whole curve, so no single cutoff is baked in; accuracy is evaluated at exactly one confusion matrix, hence one threshold.
Why can AUC be misleading on heavily imbalanced data?
FPR normalizes false positives by the huge negative count, so even many false alarms shrink to a small rate, making the curve look better than the real-world false-alarm burden.
Why do we sweep the threshold from high to low to trace the curve?
High starts strict (few positives, near ) and each drop admits more predicted positives, moving right/up until everything is flagged at — this orders points naturally.
Why is AUC called threshold-agnostic and why is that useful for model selection?
It summarizes performance without committing to an operating point, so during development you compare models fairly, then choose a deployment threshold afterward.
Edge cases
What is the ROC curve of a model that gives every example the identical score?
All points collapse — the curve is the diagonal (AUC = 0.5); with no score separation there is no way to rank one class above the other.
What happens to the curve when many examples tie at the same score?
The curve steps diagonally across that tie (a sloped segment, not a right-angle staircase), and the Mann–Whitney sum counts each tie as 0.5 to stay consistent.
What is AUC when there are zero positive examples in the test set?
TPR is undefined (division by ), so AUC is undefined — you need at least one of each class for ROC to exist.
Where does a classifier sit on the ROC plane if it flags nothing, regardless of input?
Exactly at : zero FP and zero TP; it's one valid but useless operating point on every curve.
Where does an "always predict positive" classifier land?
At : it catches all positives (TPR = 1) but also flags all negatives (FPR = 1) — the opposite useless corner.
What does a curve that dips below the diagonal in one region indicate?
In that FPR range the model ranks negatives above positives (worse than random locally); the crossing means AUC alone hides a region where flipping predictions would help.
If the test set has exactly one positive and one negative, how many curve shapes are possible?
Just two — either the positive outscores the negative (AUC = 1, corner path) or it doesn't (AUC = 0, mirror path); ties give AUC = 0.5.
Recall Fast self-check
AUC threshold-dependent? ::: No — it is computed over all thresholds simultaneously. Diagonal ROC line means? ::: Random, non-discriminating classifier, AUC = 0.5. Best metric replacement when negatives dominate? ::: PR-AUC from the Precision-Recall Curve.