Accuracy, precision, recall, F1-score — Hinglish notes

2.6.7 · AI-ML › Model Evaluation & Selection

Foundation: Confusion Matrix

Actually + Actually - Predicted + TP FP Predicted - FN TN

True Positive (TP): Model ne positive predict kiya, actually positive hai
True Negative (TN): Model ne negative predict kiya, actually negative hai
False Positive (FP): Model ne positive predict kiya, actually negative hai (Type I error)
False Negative (FN): Model ne negative predict kiya, actually positive hai (Type II error)

Yeh kyun important hai: Har classification metric in chaar numbers se compute hoti hai. Confusion matrix samajhna matlab har metric kya emphasize karti hai, yeh samajhna hai.

Metric 1: Accuracy

First principles se: Hum chahte hain $\frac{correct predictions}{total predictions}$ .

Correct predictions = TP + TN (positives sahi mile + negatives sahi mile) Total predictions = TP + TN + FP + FN (chaaron quadrants)

$Accuracy = \frac{TP + TN}{TP + TN + FP + FN}$

Interpretation: Accuracy = 0.95 matlab 95% predictions sahi hain.

$Accuracy = \frac{45 + 48}{45 + 48 + 2 + 5} = \frac{93}{100} = 0.93$

93% kyun? Kyunki 100 mein se 93 predictions reality se match ki.

Kyun sahi lagta hai: 95% impressive lagta hai — sirf 5% errors.

Problem: Fraud detection ka sochte hain jisme 1000 transactions hain: 990 legitimate, 10 fraudulent. Ek lazy model jo har cheez ke liye "legitimate" predict karta hai:

TP = 0 (koi fraud nahi pakda)
TN = 990 (saare legitimate sahi identify kiye)
FP = 0
FN = 10 (saara fraud miss)

$Accuracy = \frac{0 + 990}{1000} = 0.99 = 99%$

Fix: Imbalanced datasets pe accuracy misleading hoti hai kyunki majority class dominate karti hai. Ek useless model ki accuracy high ho sakti hai. Iski jagah precision/recall use karo.

Metric 2: Precision

First principles se: Hum chahte hain $\frac{correct positive predictions}{all positive predictions}$ .

Correct positive predictions = TP Saari positive predictions = TP + FP (actual positives + false alarms)

$Precision = \frac{TP}{TP + FP}$

Interpretation: Precision = 0.8 matlab jab model "positive" bolta hai, toh woh 80% time sahi hota hai.

Alternative naam: Positive Predictive Value (PPV)

$Precision = \frac{80}{80 + 20} = \frac{80}{100} = 0.8$

Yeh kyun matter karta hai: 20% flagged emails false alarms hain. Users important emails miss karenge! Spam filters ke liye, high precision critical hai — spam ko jaane dena better hai rather than important mail block karna.

Yeh specific numbers kyun?

TP = 80: Model ne 80 spam emails sahi identify kiye
FP = 20: Model ne 20 legitimate emails ko galti se spam flag kiya
Total predicted positive = 100 (jo model ne spam claim kiya)
Precision batata hai: "Un 100 emails mein se jo model ne spam bola, sirf 80 actually spam the"

Metric 3: Recall

First principles se: Hum chahte hain $\frac{positives we found}{all actual positives}$ .

Positives jo mile = TP Saare actual positives = TP + FN (mile positives + missed positives)

$Recall = \frac{TP}{TP + FN}$

Interpretation: Recall = 0.9 matlab model 90% actual positives pakad leta hai.

Alternative naam: Sensitivity, True Positive Rate (TPR)

$Recall = \frac{18}{18 + 2} = \frac{18}{20} = 0.9$

90% kyun? Model ne 20 actual cancer cases mein se 18 find kiye. 2 miss ho gaye (10%).

Recall yahan kyun matter karta hai: Cancer miss karna (false negative) catastrophic hai. Medical screening ke liye, high recall critical hai — false alarms hona better hai rather than disease miss karna.

Yeh specific numbers kyun?

TP = 18: 18 beemar patients sahi identify kiye
FN = 2: 2 beemar patients miss ho gaye (unka test negative aaya lekin unhe cancer tha)
Total actual positive = 20 (ground truth)
Recall batata hai: "Un 20 patients mein se jo actually cancer se beemar the, model ne 18 detect kiye"

Kyun sahi lagta hai: Dono ke numerator mein TP hota hai aur dono similar lagte hain.

Problem: Inke alag-alag denominators hain aur yeh alag-alag failure modes measure karte hain:

Precision: "Jo maine claim kiya positive tha, usme se kitna sahi tha?" (denominator = predicted positive)
Recall: "Jo actually positive tha, usme se kitna maine find kiya?" (denominator = actual positive)

Dono alag karne ke liye example:

Model sirf ek baar positive predict karta hai, aur woh sahi hota hai: TP=1, FP=0, FN=99
- Precision = 1/1 = 100% (perfect! har prediction sahi thi)
- Recall = 1/100 = 1% (terrible! 99% miss ho gaye)

Fix: Precision poochta hai "Mere alarms kitne reliable hain?", recall poochta hai "Mujhe kitne real cases mile?"

Metric 4: F1-Score

First principles se: Hum precision aur recall ka harmonic mean chahte hain, arithmetic mean nahi. Harmonic kyun?

Agar arithmetic mean use karein: $\frac{P + R}{2}$ Problem: $P = 1, R = 0.01 \to mean = 0.505$ (theek lagta hai, lekin recall terrible hai!)

Harmonic mean: $\frac{2}{\frac{1}{P} + \frac{1}{R}}$ (reciprocals average karo, phir wapas flip karo)

Yeh extreme imbalance ko penalize karta hai — agar P ya R mein se koi bhi low hai, toh F1 low hoga.

Simplify karne par: $F1 = \frac{2}{\frac{1}{P} + \frac{1}{R}} = \frac{2 P R}{P + R} = \frac{2 \cdot TP}{2 \cdot TP + FP + FN}$

Last form kyun? $P = \frac{TP}{TP + FP}$ aur $R = \frac{TP}{TP + FN}$ substitute karo:

$\frac{2 P R}{P + R} = \frac{2 \cdot \frac{TP}{TP + FP} \cdot \frac{TP}{TP + FN}}{\frac{TP}{TP + FP} + \frac{TP}{TP + FN}}$

Numerator aur denominator ko $(TP + FP) (TP + FN)$ se multiply karo:

$= \frac{2 \cdot TP ^{2}}{( TP + FN ) TP + ( TP + FP ) TP} = \frac{2 \cdot TP ^{2}}{TP ( 2 TP + FP + FN )} = \frac{2 TP}{2 TP + FP + FN}$

Model B (balanced):

TP=70, FP=20, FN=20
Precision = 70/90 = 0.778
Recall = 70/90 = 0.778
F1 = 2(0.778)(0.778)/(0.778+0.778) = 0.778

Model B ka F1 zyada kyun hai: Model A se precision kam hone ke bawajood, Model B ki balance usse better F1 deti hai. Harmonic mean balance ko reward karta hai.

Model A F1 ke liye step-by-step:

Precision compute karo: $\frac{50}{50 + 5} = \frac{50}{55} = 0.909$
Recall compute karo: $\frac{50}{50 + 40} = \frac{50}{90} = 0.556$
Harmonic mean apply karo: $\frac{2 \times 0.909 \times 0.556}{0.909 + 0.556} = \frac{1.011}{1.465} = 0.690$

Precision-Recall Tradeoff

Lower threshold → zyada baar positive predict karta hai:

Zyada true positives pakad leta hai → recall ↑
Lekin zyada false alarms bhi → precision ↓

Higher threshold → kam baar positive predict karta hai:

Kam false alarms → precision ↑
Lekin zyada true positives miss ho jaate hain → recall ↓

Tum dono simultaneously maximize nahi kar sakte (trivially 100% perfect classification ke alawa). Isliye F1-score exist karta hai — ek reasonable balance dhundhne ke liye.

Dataset: 10 samples (5 actually positive, 5 actually negative) Scores: [0.9, 0.8, 0.6, 0.4, 0.3] (positives), [0.7, 0.5, 0.4, 0.2, 0.1] (negatives)

Threshold = 0.5:

Predicted positive: 0.9, 0.8, 0.7, 0.6, 0.5
TP=4, FP=2, FN=1
Precision = 4/6 = 0.667
Recall = 4/5 = 0.8

Threshold = 0.75:

Predicted positive: 0.9, 0.8
TP=2, FP=0, FN=3
Precision = 2/2 = 1.0
Recall = 2/5 = 0.4

Kyun? Higher threshold → kam predictions → kam false positives (precision up) lekin zyada missed positives (recall down).

Kaunsi Metric Kab Use Karein

Scenario	Use Karo	Kyun
Balanced classes, equal error costs	Accuracy	Saari errors equally matter karti hain
Imbalanced classes	Accuracy kabhi nahi	Majority class dominate karta hai
False positives costly hain (spam filter, drug approval)	Precision	False alarms afford nahi kar sakte
False negatives costly hain (disease screening, fraud detection)	Recall	Cases miss karna afford nahi kar sakte
Ek number chahiye, balanced concern	F1-Score	Dono ka harmonic mean
P/R ke liye alag costs hain	F-beta score	Weighted harmonic mean

F-beta score F1 ko generalize karta hai: $F_{β} = (1 + β^{2}) \frac{P R}{β ^{2} P + R}$

$β < 1$ : precision ko favor karta hai
$β > 1$ : recall ko favor karta hai
$β = 1$ : F1-score (equal weight)

Multi-Class Extension

Multi-class classification (>2 classes) ke liye, metrics per class compute karo aur phir aggregate karo:

Macro-average: Har class ke liye metric compute karo, phir average karo (saari classes ko equally treat karta hai)
Micro-average: Saari classes ke TP/FP/FN pool karo, phir metric compute karo (saare samples ko equally treat karta hai)
Weighted-average: Har class ko uski frequency se weight karo

Example intuition: 3 classes (A: 100 samples, B: 50 samples, C: 10 samples)

Macro: (metric_A + metric_B + metric_C)/3 (class C utna hi count karta hai jitna A)
Micro: pehle saare TP/FP/FN aggregate karo (class A size ki wajah se dominate karta hai)
Weighted: 100, 50, 10 se weight karo (balanced approach)

Recall Ek 12-Saal-Ke Bachche Ko Samjhao

Socho tum ek teacher ho jo True/False tests grade kar rahe ho, lekin tum measure karna chahte ho ki tum un bacchon ko pakadne mein kitne ache ho jo guess kar rahe hain.

Accuracy aise hai jaise "Saare sawaalon mein se maine kitne sahi grade kiye?" Agar 95 sawaalon ka jawab FALSE hai aur sirf 5 ka TRUE hai, aur main sab kuch FALSE mark kar doon, toh mujhe 95% accuracy milti hai — lekin maine TRUE wale sawaalon ke baare mein kuch nahi seekha!

Precision hai "Jab main kisi paper ko circle karke kahoon 'Yeh bachcha guess kar raha tha!', toh main kitni baar sahi hota hoon?" Agar main 10 bacchon par ilzaam lagaoon aur 8 actually guess kar rahe the, toh yeh 80% precision hai. 2 jinhein maine galti se accuse kiya woh false alarms hain.

Recall hai "Un saare bacchon mein se jo actually guess kar rahe the, maine kitne pakde?" Agar 20 bachche guess kar rahe the aur maine sirf 15 pakde, toh yeh 75% recall hai. Main 5 miss kar gaya.

F1 aise hai jaise ek report card jo dono combine karta hai: "Kya main guessers ko pakadne mein achi hoon AUR innocent bacchon par ilzaam na lagane mein?" Agar main ek mein great hoon lekin doosre mein terrible, toh mera F1 low hoga. Mujhe balanced rehna hoga.

Tradeoff: Agar main SABHI par guess karne ka ilzaam lagaoon (saare guessers pakadne ke liye), toh mera recall perfect hoga lekin precision terrible (bahut zyada false accusations). Agar main sirf tab accuse karoon jab 100% sure hoon, toh meri precision perfect hogi lekin recall terrible (zyatar guessers miss ho jaenge). F1 sweet spot dhundhne mein help karta hai.

Denominator trick:

Precision: denominator mein P hai (predicted positive = TP + FP)
Recall: denominator mein N hai (actual positive = TP + FN, jahan FN = False Negative)

Harm mnemonic:

Precision: false alarms se harm minimize karo (denominator mein FP)
Recall: misses se harm minimize karo (denominator mein FN)

Connections

Confusion Matrix - saari classification metrics ki foundation
ROC Curve and AUC - saare thresholds pe precision-recall tradeoff visualize karta hai
Cross-Validation - in metrics ko use karo models fairly compare karne ke liye
Class Imbalance Handling - kyun accuracy fail hoti hai aur sampling/weighting kyun help karta hai
Cost-Sensitive Learning - jab false positives aur false negatives ke alag-alag costs hoon
Multi-Label Classification - in metrics ko multiple labels per sample tak extend karna
Threshold Optimization - apne use case ke liye best decision threshold dhundhna
Precision-Recall Curve - imbalanced datasets ke liye ROC ka alternative

#flashcards/ai-ml

Confusion matrix mein chaar outcomes kya hain? :: True Positive (predicted +, actually +), True Negative (predicted -, actually -), False Positive (predicted +, actually -), False Negative (predicted -, actually +)

Accuracy formula first principles se derive karo :: Accuracy = (correct predictions)/(total predictions) = (TP + TN)/(TP + TN + FP + FN), jahan correct = positives sahi mile + negatives sahi mile

Imbalanced datasets pe accuracy misleading kyun hoti hai?

Majority class dominate karta hai. Example: 99% legitimate transactions, ek model jo sab kuch "legitimate" predict kare woh 99% accuracy paata hai lekin zero fraud pakdta hai (useless).

Precision formula derive karo. Yeh kya measure karta hai?

Precision = TP/(TP + FP). Measure karta hai "Meri positive predictions mein se kitni sahi thi?" High precision = kam false alarms.

Recall formula derive karo. Yeh kya measure karta hai?

Recall = TP/(TP + FN). Measure karta hai "Saare actual positives mein se kitne maine find kiye?" High recall = kam misses.

Precision-recall tradeoff kya hai?

Decision threshold kam karna recall badhata hai (zyada positives pakad leta hai) lekin precision ghata deta hai (zyada false alarms). Threshold badhana opposite karta hai. Dono simultaneously maximize nahi kar sakte.

F1 ke liye arithmetic mean ki jagah harmonic mean kyun use karte hain?

Harmonic mean extreme imbalance ko penalize karta hai. Agar P=1.0 aur R=0.01, arithmetic mean = 0.505 (misleading), harmonic mean (F1) = 0.02 (sahi se poor performance dikhata hai).

F1-score formula derive karo :: F1 = 2/(1/P + 1/R) = 2PR/(P+R) = 2TP/(2TP + FP + FN). Yeh precision aur recall ka harmonic mean hai.

Recall pe precision kab prioritize karni chahiye?

Jab false positives costly hoon. Examples: spam filtering (important emails block nahi kar sakte), drug approval (unsafe drugs approve nahi kar sakte), content moderation (innocent users ban nahi kar sakte).

Precision pe recall kab prioritize karni chahiye?

Jab false negatives costly hoon. Examples: disease screening (cancer miss nahi kar sakte), fraud detection (fraudulent transactions miss nahi kar sakte), security threats (intrusions miss nahi kar sakte).

Multi-class metrics ke liye macro aur micro averaging mein kya difference hai?

Macro: har class ke liye metric compute karo, phir average karo (saari classes ko equally treat karta hai). Micro: pehle saari classes ke TP/FP/FN pool karo, phir metric compute karo (saare samples ko equally treat karta hai, majority class dominate karta hai).

Decision threshold badhane se precision aur recall pe kya effect hota hai?

Higher threshold → kam baar positive predict karta hai → precision badhta hai (kam false alarms), recall ghatta hai (zyada misses). Lower threshold → opposite effect.

F-beta score kya hai aur β≠1 kab use karte hain?

F_β = (1+β²)PR/(β²P + R). β<1 use karo precision favor karne ke liye, β>1 use karo recall favor karne ke liye, β=2 (F2) recall-focused tasks jaise search engines ke liye common hai.