Precision-recall tradeoff and curves — Hinglish notes

2.6.8 · AI-ML › Model Evaluation & Selection

Overview

Precision-recall tradeoff inverse relationship describe karta hai precision aur recall ke beech jab hum classifier ka decision threshold adjust karte hain. Is tradeoff ko samajhna imbalanced classification problems ke liye critical hai jahan accuracy misleading hoti hai (fraud detection, disease diagnosis, spam filtering).

Yeh kyun important hai: ROC curves ki tarah nahi jo imbalanced datasets par overly optimistic ho sakti hain, PR curves directly dikhati hain ki true positives pakadne ke liye aap kitne false alarms accept karte ho.

Figure — Precision-recall tradeoff and curves

Core Concepts

Definition Precision aur Recall (First Principles)

Ek binary classifier ko consider karo jiske predictions $\overset{y}{^}$ hain aur true labels $y$ hain: Confusion Matrix: | | Predicted Positive | Predicted Negative | |-------------------|--------------------| | Actually Positive | TP (True Positive) | FN (False Negative) | | Actually Negative | FP (False Positive) | TN (True Negative) |

$Precision = \frac{T P}{T P + F P}$ Yeh kya measure karta hai: Un sab items mein se jo humne positive flag kiye, unka kitna fraction actually positive hai?
Intuition: "Jab main 'haan' kehta hoon, toh kitni baar main sahi hota hoon?"

$Recall = \frac{T P}{T P + F N}$ Yeh kya measure karta hai: Saare actually positive items mein se, kitne fraction ko humne successfully pakad liya?
Intuition: "Haystack mein jitni suiyaan thi, unme se kitni maine dhundh li?"

Formula Decision Threshold Se Tradeoff Derive Karna

Setup: Ek classifier ek score $s (x) \in [0, 1]$ output karta hai (jaise probability). Hum positive predict karte hain agar $s (x) \geq τ$ (threshold).

Step 1 — Jab $τ$ decrease hota hai toh kya hota hai?

Zyada samples threshold cross karte hain → zyada predicted positives
$T P$ badhta hai (hum zyada actual positives pakad lete hain) → recall increases
$F P$ bhi badhta hai (hum zyada negatives ko galat flag karte hain) → precision decreases Step 2 — Inverse relationship kyun hai? $Precision = \frac{T P}{T P + F P}, Recall = \frac{T P}{T P + F N}$

Jab $τ ↓$ hota hai:

Recall ka numerator ( $T P$ ) badhta hai → recall ↑
Precision ka denominator ( $T P + F P$ ) numerator se zyada tez badhta hai agar nayi predictions mein bahut se false positives hain → precision ↓

Step 3 — Precision-Recall Curve

Saare thresholds $τ \in [0, 1]$ ke liye $(r, p)$ points plot karo:

X-axis: Recall $r$
Y-axis: Precision $p$ Jaise recall badhta hai (left to right), precision generally decrease hoti hai. Curve ki shape model quality reveal karti hai.

Worked example Worked Example 1: Manual Threshold Sweep

Scenario: Fraud detection. 10 transactions, classifier scores:

Transaction	Score	True Label
1	0.95	Fraud
2	0.88	Fraud
3	0.75	Normal
4	0.65	Fraud
5	0.55	Normal
6	0.45	Normal
7	0.30	Normal
8	0.20	Fraud
9	0.10	Normal
10	0.05	Normal

True fraud count = 4 (transactions 1, 2, 4, 8).

Threshold $τ = 0.80$ :

Predicted positive: transactions with score ≥ 0.80 → {1, 2}
TP = 2 (dono fraud hain), FP = 0, FN = 2 (4, 8 miss ho gaye)
Precision = $\frac{2}{2 + 0} = 1.00$ ✅ Perfect confidence
Recall = $\frac{2}{2 + 2} = 0.50$ ⚠️ Aadhe miss ho gaye

Yeh step kyun?: High threshold = sirf confident predictions → high precision, low recall.

Threshold $τ = 0.60$ :

Predicted positive: {1, 2, 3, 4}
TP = 3 (1, 2, 4 fraud hain), FP = 1 (3 normal hai), FN = 1 (8 miss ho gaya)
Precision = $\frac{3}{3 + 1} = 0.75$
Recall = $\frac{3}{3 + 1} = 0.75$

Yeh step kyun?: Lower threshold transaction 3 (normal) ko include karta hai → precision girta hai, recall badhta hai.

Threshold $τ = 0.15$ :

Predicted positive: {1, 2, 3, 4, 5, 6, 7, 8}
TP = 4 (saara fraud pakad liya), FP = 4 (3, 5, 6, 7), FN = 0
Precision = $\frac{4}{4 + 4} = 0.50$
Recall = $\frac{4}{4 + 0} = 1.00$ ✅ Sab pakad liya

Yeh step kyun?: Bahut low threshold → bahut false alarms → precision 50% tak gir jaati hai, lekin perfect recall.

Tradeoff visualized:

$τ = 0.80$ : (Recall=0.50, Precision=1.00)
$τ = 0.60$ : (Recall=0.75, Precision=0.75)
$τ = 0.15$ : (Recall=1.00, Precision=0.50)

Jaise recall badhta hai, precision decrease hoti hai.

Worked example Worked Example 2: Area Under PR Curve (AUC-PR)

AUC-PR kyun?: Curve quality ko summarize karne wala ek single number. Higher = better across all thresholds.

First principles se derivation: $AUC-PR = \int_{0}^{1} p (r) d r$

Jahaan $p (r)$ precision hai recall ke function ke roop mein. Discrete approximation (trapezoidal rule): $AUC-PR \approx \sum_{i = 1}^{n - 1} \frac{( p _{i} + p _{i + 1} )}{2} \cdot (r_{i + 1} - r_{i})$

3 threshold points wala example:

Threshold	Recall	Precision
High	0.2	1.0
Medium	0.6	0.8
Low	1.0	0.5

Step 1: Increasing recall ke hisaab se sort karo (already sorted hai).

Step 2: Trapezoid formula apply karo: $AUC-PR = \frac{( 1.0 + 0.8 )}{2} \cdot (0.6 - 0.2) + \frac{( 0.8 + 0.5 )}{2} \cdot (1.0 - 0.6)$

$= \frac{1.8}{2} \cdot 0.4 + \frac{1.3}{2} \cdot 0.4$

$= 0.9 \cdot 0.4 + 0.65 \cdot 0.4 = 0.36 + 0.26 = 0.62$

Yeh step kyun?: Trapezoids points ko connect karne wali curve ke neeche ka area approximate karte hain.

Interpretation: AUC-PR = 0.62 matlab classifier recall levels across decent precision maintain karta hai. Perfect classifier = 1.0, random baseline ≈ (fraction of positives).

Common mistake Common Mistake: PR Curve Ko ROC Curve Ki Tarah Treat Karna

Galat approach: "High AUC-ROC matlab high AUC-PR."

Yeh sahi kyun lagta hai: Dono threshold-based curves hain jo classifier quality measure karti hain.

Yeh galat kyun hai:

ROC False Positive Rate (FPR = $\frac{F P}{F P + T N}$ ) use karta hai, jo true negatives par depend karta hai
Imbalanced datasets mein (jaise 1% fraud), TN bahut bada hota hai → FPR chhota rehta hai even with many FP → ROC acha dikhta hai
PR curve Precision = $\frac{T P}{T P + F P}$ use karta hai, jo TN ko ignore karta hai → FP ke liye sensitive hai even jab negatives abundant hain

Example: 1000 samples, 10 positive, 990 negative.

Model 100 positives predict karta hai: TP=8, FP=92, TN=898, FN=2
ROC: TPR=0.8, FPR= $\frac{92}{990}$ =0.093 → decent lagta hai
PR: Precision= $\frac{8}{100}$ =0.08, Recall=0.8 → terrible precision!

Fix: Imbalanced problems ke liye PR curves use karo. ROC curves misleading ho sakti hain jab negatives dominate karte hain.

Recall Ek 12 Saal Ke Bachche Ko Explain Karo

Imagine karo tum ek lifeguard ho jo crowded pool mein doobte swimmers dhundh rahe ho. Tumhare paas binoculars hain jo zoom in kar sakte hain, lekin zyada zoom karne par tum sirf ek choti si area dekh sakte ho.

Precision = jab tum "doob raha hai!" chillate ho, toh kitni baar koi actually doob raha hota hai?
Recall = un sab logon mein se jo actually doob rahe the, kitno ko tumne spot kiya?

Agar tum har baar chillate ho jab koi paani mein splash karte dekho (low threshold):

Tum har doobne wale ko pakad loge (high recall ✅)
Lekin tum khelte huye bacchon par bhi chillaoге (low precision ❌)

Agar tum sirf tab chillate ho jab 100% sure ho (high threshold):

Har koi jise tum call out karte ho actually doob raha hota hai (high precision ✅)
Lekin tum kuch ko miss kar sakte ho (low recall ❌)

Precision-recall curve yahi tradeoff dikhati hai: jaise tum zyada doobne wale pakadne ki koshish karte ho (recall up), tum zyada mistakes karte ho (precision down). Tum decide karte ho ki apna "chillane ka threshold" kahaan set karna hai is baat ke hisaab se ki kya zyada important hai: kisi ko miss na karna, ya false alarms na karna.

Connections

2.6.01-Confusion-matrix-and-basic-metrics — Precision aur recall confusion matrix cells se derive hote hain
2.6.07-ROC-curves-and-AUC — Imbalanced datasets ke liye ROC vs PR curves
2.6.09-F1-score-and-Fbeta-score — Single-metric optimization ke liye precision aur recall ka harmonic mean
2.6.12-Class-imbalance-handling — Imbalanced problems ke liye PR curves kyun critical hain
3.2.04-Decision-thresholds-and-calibration — PR curve par optimal threshold kaise choose karein
4.1.08-Cost-sensitive-learning — Business costs ke hisaab se precision vs recall weigh karna

Flashcards

#flashcards/ai-ml

Precision-recall tradeoff kya hai?

Jab tum recall badhane ke liye decision threshold lower karte ho (zyada true positives pakadne ke liye), precision decrease hoti hai kyunki tum zyada false positives bhi flag karte ho. Dono ka inverse relationship hota hai.

Imbalanced datasets ke liye PR curves ROC curves se better kyun hain?

PR curves precision (TP/(TP+FP)) use karti hain, jo true negatives ko ignore karti hai. Imbalanced data mein bahut zyada negatives hone par, ROC ka FPR bahut false positives hone par bhi low rehta hai, jisse model actual se better lagta hai. PR curves false positives ke liye sensitive hain.

AUC-PR ke liye no-skill baseline kya hai?

Dataset mein positive samples ka fraction. Jaise agar data ka 5% positive hai, toh random classifier ka AUC-PR ≈ 0.05 hoga, 0.5 nahi.

Confusion matrix se precision kaise calculate karte hain?

Precision = TP / (TP + FP), predicted positives ka wo fraction jo actually positive hai.

Confusion matrix se recall kaise calculate karte hain?

Recall = TP / (TP + FN), actual positives ka wo fraction jo successfully positive predict kiya gaya.

Agar decision threshold increase karo toh precision aur recall ka kya hoga?

Precision badhti hai (zyada confident predictions mein kam false positives), recall decrease hoti hai (bahut strict hone par tum kuch true positives miss kar dete ho).

PR curve par (Recall=1.0, Precision=0.3) point ka kya matlab hai?

Classifier saare true positives pakad leta hai (perfect recall) lekin bahut false positive errors karta hai (sirf 30% positive predictions sahi hain).

AUC-PR kaise calculate karte hain?

0 se 1 tak recall par precision integrate karo, ya trapezoidal approximation use karo: consecutive (recall, precision) points ke beech trapezoid areas ka sum.

Threshold lower karne se recall kyun badhta hai?

Zyada samples threshold cross karte hain aur positive predict hote hain, isliye tum zyada true positives pakad lete ho (TP badhta hai), aur recall = TP/(TP+FN) badhta hai.

Ek typical PR curve ki shape kaisi hoti hai?

Recall badhne par precision decrease hoti hai (inverse relationship), aksar high recall par steep drop hota hai jab model bahut zyada false positive errors karna shuru kar deta hai.

Overview

Core Concepts

Connections

Flashcards

Concept Map