ROC curve and AUC — Hinglish notes

2.6.9 · AI-ML › Model Evaluation & Selection

The Core Concept

Ek Receiver Operating Characteristic (ROC) curve ek binary classifier ki performance ko plot karta hai jab hum iska decision threshold 0 se 1 tak sweep karte hain. Y-axis pe: True Positive Rate (TPR), jise sensitivity ya recall bhi kehte hain. X-axis pe: False Positive Rate (FPR).

YE AXES KYUN?

TPR jawab deta hai: "Saare actual positives mein se, humne kitna fraction pakda?" (Hum chahte hain ye HIGH ho)
FPR jawab deta hai: "Saare actual negatives mein se, kitna fraction humne galti se flag kiya?" (Hum chahte hain ye LOW ho)

Jab tum threshold lower karte ho ("positive" zyada baar predict karte ho), DONO rates badhti hain—lekin alag-alag speeds pe, depending on tumhara model kitna acha hai.

YE INTEGRAL KYUN? AUC measure karta hai ki tumhara classifier random guessing ke upar saare operating points pe kitna total "lift" provide karta hai. Ye us probability ke barabar hai ki ek randomly chuna hua positive example ek randomly chune gaye negative example se zyada rank kare.

ROC Curve ko First Principles se Derive Karna

Setup: Tumhare paas ek trained binary classifier hai jo examples $i = 1, \dots, n$ ke liye scores $s_{i} \in [0, 1]$ output karta hai. True labels $y_{i} \in {0, 1}$ hain. Maano $n_{+} = \sum_{i} 1 [y_{i} = 1]$ aur $n_{-} = \sum_{i} 1 [y_{i} = 0]$ .

Step 1: Ek threshold $t$ chuno. Predict karo $\overset{y}{^}_{i} = 1$ agar $s_{i} \geq t$ , warna $\overset{y}{^}_{i} = 0$ .

Step 2: Confusion matrix entries count karo. $T P (t) = \sum_{i : y_{i} = 1} 1 [s_{i} \geq t]$ $F P (t) = \sum_{i : y_{i} = 0} 1 [s_{i} \geq t]$ $F N (t) = \sum_{i : y_{i} = 1} 1 [s_{i} < t] = n_{+} - T P (t)$ $T N (t) = \sum_{i : y_{i} = 0} 1 [s_{i} < t] = n_{-} - F P (t)$

YE COUNTING KYUN? Har predicted positive jiska true label 1 hai wo TP hai; har predicted positive jiska true label 0 hai wo FP hai. Threshold decide karta hai ki kaun si predictions positive hain.

Step 3: Rates compute karo. $TPR (t) = \frac{T P ( t )}{n _{+}}, FPR (t) = \frac{F P ( t )}{n _{-}}$

Step 4: $t$ ko 1 se 0 tak vary karo.

$t = 1$ pe: Koi bhi prediction positive nahi → $T P = 0, F P = 0$ → point $(0, 0)$
$t = 0$ pe: Saari predictions positive hain → $T P = n_{+}, F P = n_{-}$ → point $(1, 1)$
Tumhare dataset mein har unique score ek naya point ROC curve pe create karta hai.

$t$ lower karne se hum upar aur daayein kyun jaate hain? Jaise-jaise $t$ decrease hota hai, $1 [s_{i} \geq t]$ zyada examples ke liye 1 ban jaata hai. TP aur FP dono counts badhte hain (kabhi nahi ghatate). Isliye TPR aur FPR dono monotonically increase karte hain.

AUC Values ko Interpret Karna

AUC Range	Interpretation
0.9 - 1.0	Excellent discrimination
0.8 - 0.9	Good discrimination
0.7 - 0.8	Acceptable
0.5 - 0.7	Poor (random se better, lekin barely useful)
0.5	Random classifier (diagonal line)
< 0.5	Random se bhi bura (tum predictions flip karke >0.5 pa sakte ho)

0.5 baseline KYUN hai? Ek random classifier scores uniformly at random assign karta hai. On average, ye ek positive ko negative se zyada 50% time rank karta hai.

Worked example Worked Example 1: ROC Hand se Compute Karna

Scenario: 8 examples scores aur true labels ke saath:

Example	Score	True Label
A	0.9	1
B	0.8	1
C	0.7	0
D	0.6	1
E	0.55	0
F	0.5	1
G	0.3	0
H	0.1	0

$n_{+} = 4, n_{-} = 4$

Step 1: Score ke hisaab se descending sort karo (already done).

Step 2: Har unique threshold ke liye scores ke beech TP aur FP count karo: | Threshold $t$ | Positives (score ≥ t) | TP | FP | TPR = TP/4 | FPR = FP/4 | |------------|----------------------|----|--------| | 1.0 | none | 0 | 0 | 0.00 | 0.00 | | 0.9 | A | 1 | 0.25 | 0.00 | | 0.8 | A,B | 2 | 0.50 | 0.00 | | 0.7 | A,B,C | 2 | 1 | 0.50 | 0.25 | | 0.6 | A,B,C,D | 3 | 1 | 0.75 | 0.25 | | 0.55 | A,B,C,D,E | 3 | 2 | 0.75 | 0.50 | | 0.5 | A,B,C,D,E,F | 4 | 2 | 1.00 | 0.50 | | 0.3 | all but H | 4 | 3 | 1.00 | 0.75 | | 0.0 | all | 4 | 4 | 1.00 | 1.00 |

YE STEP KYUN? Example C (score 0.7, label 0) humara pehla false positive hai. Usse pehle, hum do true positives (A, B) zero false alarms ke saath pakadh lete hain.

Step 3: In (FPR, TPR) points ko plot karo aur connect karo. ROC curve jaata hai: $(0, 0) \to (0, 0.25) \to (0, 0.5) \to (0.25, 0.5) \to (0.25, 0.75) \to (0.5, 0.75) \to (0.5, 1.0) \to (0.75, 1.0) \to (1.0, 1.0)$

Step 4: Trapezoids se AUC compute karo: $AUC = \sum_{i} \frac{1}{2} (FPR_{i + 1} - FPR_{i}) (TPR_{i} + TPR_{i + 1})$ $= \frac{1}{2} (0.25) (0 + 0.5) + \frac{1}{2} (0.25) (0.5 + 0.75) + \frac{1}{2} (0.25) (0.75 + 1.0) + \frac{1}{2} (0.25) (1.0 + 1.0)$ $= 0.0625 + 0.15625 + 0.21875 + 0.25 = 0.6875$

Interpretation: AUC = 0.69 → acceptable lekin great nahi. Model positives ko negatives se zyada baar rank karta hai, lekin substantial overlap hai.

Worked example Worked Example 2: Do Models Compare Karna

Model A: Tumor size ke basis pe cancer predict karta hai. AUC = 0.78 Model B: Tumor size + patient age + genetic markers ke basis pe cancer predict karta hai. AUC = 0.91

AUC comparison yahan valid KYUN hai? Dono models same test set pe same outcome predict karte hain. AUC threshold-agnostic hai, isliye hum comparison ko ek model ke liye favorable threshold pick karke bias nahi kar rahe.

Decision: Model B clearly better hai cancer ko non-cancer se saare operating points pe alag karne mein. Chahe baad mein tumhe high-precision threshold (low FPR) ya high-recall threshold (high TPR) chahiye, Model B Model A se behtar perform karega.

YE KYUN MATTER KARTA HAI: Medical screening mein, tum high sensitivity pe deploy kar sakte ho (saare cancers pakdo, false alarms tolerate karo) lekin development ke dauran best model architecture chunne ke liye AUC use karo.

Common mistake Common Mistake 2: Imbalanced Classes ke liye AUC Bina Caution ke Use Karna

Galat idea: "AUC imbalanced datasets ke liye perfect hai kyunki ye threshold-agnostic hai." Kyun sahi lagta hai: FPR aur TPR apne class sizes se normalize hote hain (FP/n₋, TP/n₊), toh class imbalance "account for" ho jaata hai.

Fix: AUC severely imbalanced data pe overly optimistic ho sakta hai. Agar negatives positives se bahut zyada hain, toh high FPR (absolute terms mein, bahut false alarms) phir bhi low FPR rate mein translate hota hai. Curve acha lagta hai, lekin practice mein tum false positives mein dube ho. Better approach: Precision-Recall AUC bhi compute karo (precision-recall curve ke neeche ka area), jo FP ko huge negative class se normalize NAHI karta. Fraud detection (0.1% fraud rate) ke liye, PR-AUC ROC-AUC se zyada informative hai.

Common mistake Common Mistake 3: AUC ko Ek Maatra Metric Maanna

Galat idea: "Model X ka AUC sabse zyada hai, isliye ye best model hai."

Kyun sahi lagta hai: AUC ek single, comprehensive number hai.

Fix: AUC tumhe ranking quality ke baare mein batata hai lekin calibration, fairness, ya tumhare deployment threshold pe performance ke baare mein nahi.

Ek model jiska AUC = 0.95 hai lekin wo probabilities {0.49, 0.51} output karta hai poorly calibrated hai (tum probabilities pe trust nahi kar sakte).
Ek model ke do subgroups ke liye equal AUC ho sakta hai lekin tumhare chosen threshold pe wildly different FPR (fairness issue). Best practice: Models compare karne ke liye AUC use karo, phir chosen model ko threshold-specific metrics (precision, recall, F1) aur calibration plots se validate karo.

Doosre Metrics se Connections

ROC curve kai concepts ko unify karta hai:

Precision-Recall Curve: Imbalanced data ke liye alternative visualization (TPR vs. FPR ki jagah Precision vs. Recall).
Confusion Matrix: ROC curve bahut saare thresholds pe confusion matrices se bana hota hai.
Sensitivity and Specificity: TPR = Sensitivity; (1 - FPR) = Specificity. ROC sensitivity vs. (1 - specificity) plot karta hai.
F1-Score: Fixed threshold pe depend karta hai. ROC saare thresholds explore karta hai. Apne deployed threshold pe F1 use karo, model chunne ke liye AUC.
Cost-Sensitive Learning: Agar false positives aur false negatives ke alag-alag costs hain, toh ROC point dhundho jo expected cost minimize kare.
Multi-Class Classification: Har class ke liye one-vs-rest ROC curves tak extend karo, ya macro-average AUC use karo (classes mein average AUC).

Recall Ek 12-Saal ke Bacche ko Samjhao

Socho tum ek teacher ho jo essays ko "good" ya "bad" grade kar rahe ho ek scoring robot use karke. Robot har essay ko 0 se 100 tak score deta hai. Tumhe ek cutoff chunna hai: "70 se upar good hai, neeche bad."

Lekin agar tum sure nahi ho ki 70 sahi cutoff hai? Shayad 60 zyada good essays pakad le (tum koi miss nahi karna chahte!), lekin ye kuch bad essays ko bhi accidentally good bol deta hai.

ROC curve har ek possible cutoff try karne jaisa hai—60, 65, 70, 75, 80—aur dekhna kya hota hai. Y-axis dikhata hai "maine kitne good essays pakde?" (zyada better hai). X-axis dikhata hai "kitne bad essays ko maine accidentally good bola?" (kam better hai).

Ek perfect robot saare good essays bina kisi mistake ke pakad leta—uska curve seedha left side se upar jaata, phir top se across. Ek robot jo sirf randomly guess karta uski diagonal line hoti (50% good ones pakdega, lekin 50% bad ones ko bhi good bolega).

AUC us curve ke neeche ka area hai. Ye ek number hai jo tumhe batata hai: "overall, saare cutoffs mein, ye robot good essays ko bad ones se kitna acha pehchaan sakta hai?" AUC = 1.0 perfect hai, 0.5 random guessing hai.

ROC vs. PR Curve Kab Use Karein

Scenario	ROC-AUC Prefer Karo	PR-AUC Prefer Karo
Balanced classes (≈50/50)	✓	Either
Imbalanced (rare positives)	✗ (misleading ho sakta hai)	✓
FP aur FN ki equally care	✓
Positive class ki ZYADA care		✓
Threshold-agnostic comparison chahiye	✓	✓

KYUN? Precision = TP/(TP+FP) explicitly FP ko numerator ke context mein include karta hai. Jab positives rare hote hain, toh low FPR ka matlab bhi bahut saare absolute false positives ho sakta hai, precision ko tanking. PR curves ye expose karte hain; ROC curves ise hide kar sakte hain.

#flashcards/ai-ml

ROC curve apne axes pe kya plot karta hai? :: Y-axis: True Positive Rate (TPR), X-axis: False Positive Rate (FPR). Ye saare thresholds mein sensitivity aur false alarms ke beech tradeoff dikhata hai.

AUC kya hai aur ye kya measure karta hai?

Area Under the ROC Curve. Ye us probability ko measure karta hai ki ek randomly chuna hua positive example ek randomly chune gaye negative example se zyada rank kare. Equivalently, ye classifier ki separation ability hai jo saare thresholds pe aggregate ki gayi hai.

AUC = 0.5 ek bekaar classifier ke liye baseline KYUN hai?

Ek random classifier positive ko negative ke upar exactly 50% time rank karta hai (koi separation ability nahi). Uski ROC curve (0,0) se (1,1) tak diagonal line hai, area = 0.5 ke saath.

AUC = 0.92 ko tum kaise interpret karoge?

Excellent discrimination. Model 92% (positive, negative) pairs ko correctly rank karta hai. Ye dono classes ko saare operating points pe acha alag karta hai.

Decision threshold lower karne se TPR aur FPR ka kya hota hai?

DONO badhte hain. Threshold lower karna matlab "positive" zyada baar predict karna, isliye tum zyada true positives pakad lete ho (TPR up) lekin zyada false positives bhi flag karte ho (FPR up).

Highly imbalanced datasets pe AUC kyun misleading ho sakta hai?

FPR = FP/n₋. Jab n₋ huge hota hai, toh bahut saare absolute false positives bhi low FPR yield karte hain, ROC curve ko great dikhata hai. Lekin practice mein, tum false alarms mein dube ho. Precision-Recall AUC ise better expose karta hai.

AUC ki probabilistic interpretation kya hai? :: AUC = P(score of random positive > score of random negative). Ye saare (pos, neg) pairs ka fraction hai jahan positive zyada score karta hai.

Agar ek model ki 90% accuracy hai lekin AUC = 0.55, toh kya ho raha hai?

Model shayad majority class hamesha predict karta hai (class imbalance se high accuracy) lekin classes alag karne ki near-zero ability hai (AUC ≈ random). Ye minority class identify karne ke liye bekaar hai.

ROC-AUC aur PR-AUC mein kya fark hai?

ROC-AUC TPR vs. FPR use karta hai (dono apne class se normalize hote hain). PR-AUC Precision vs. Recall use karta hai (Precision = TP/(TP+FP), absolute FP count ke liye sensitive). PR-AUC imbalanced data ke liye better hai.

ROC curve point (0, 0) pe kab pahunchti hai?

Jab threshold = 1.0 (kuch bhi positive predict nahi). TP = 0, FP = 0, isliye TPR = 0, FPR = 0.

ROC curve point (1, 1) pe kab pahunchti hai? :: Jab threshold = 0 (sab kuch positive predict karo). TP = n₊, FP = n₋, isliye TPR = 1, FPR = 1.

ROC-AUC threshold-agnostic KYUN hai?

Ye performance ko SAARE possible thresholds pe aggregate karta hai har threshold ke liye (FPR, TPR) plot karke aur resulting curve ke neeche ka area measure karke. Tum ek single threshold pe commit nahi kar rahe.

"ROC curve top-left corner ko hug karti hai" ka matlab kya hai?

Model high TPR (y-axis) achieve karta hai low FPR (x-axis) ke saath, matlab ye zyaatar positives pakad leta hai jabki few false alarms trigger karta hai. Ye ideal classifier behavior hai.

AUC ka Mann-Whitney U statistic se kya relation hai?

Ye equivalent hain. AUC = (U / (n₊ × n₋)) jahan U un pairs ko count karta hai jahan positive > negative. Ye rank-sum interpretation ke barabar hai.