R-squared and adjusted R-squared — Hinglish notes

2.2.8 · AI-ML › Linear & Logistic Regression

What is R-squared?

Range: $R^{2} \in [0, 1]$ typical regression ke liye (intercept ke bina bure models ke liye negative bhi ho sakta hai)

Derivation from First Principles

YEH KYUN CHAHIYE? Mean $\overset{y}{ˉ}$ hamara baseline predictor hai—agar kuch nahi pata, toh hum mean predict karenge. Ek regression model yahi koshish karta hai ki iss baseline ko haraaye. Hume ek aisi metric chahiye jo compare kare ki kitna variance bacha (residuals) versus original variance.

Variance decomposition se shuru karo: $S S_{t o t} = S S_{r e g} + S S_{r es}$

YEH KYUN KAAM KARTA HAI: Total variance = Explained variance + Unexplained variance

Total variance $S S_{t o t}$ : Kitna $y$ values apne mean ke around failte hain $S S_{t o t} = \sum_{i = 1}^{n} (y_{i} - \overset{y}{ˉ})^{2}$
Residual variance $S S_{r es}$ : Predictions kitna actual values se chookti hain $S S_{r es} = \sum_{i = 1}^{n} (y_{i} - \overset{y}{^}_{i})^{2}$
Explained variance $S S_{r e g}$ : Model ki predictions mean se kitna vary karti hain $S S_{r e g} = \sum_{i = 1}^{n} (\overset{y}{^}_{i} - \overset{y}{ˉ})^{2}$

Ab $R^{2}$ derive karo: $S S_{t o t} = S S_{r e g} + S S_{r es}$ $\frac{S S _{t o t}}{S S _{t o t}} = \frac{S S _{r e g}}{S S _{t o t}} + \frac{S S _{r es}}{S S _{t o t}}$ $1 = \frac{S S _{r e g}}{S S _{t o t}} + \frac{S S _{r es}}{S S _{t o t}}$ $\frac{S S _{r e g}}{S S _{t o t}} = 1 - \frac{S S _{r es}}{S S _{t o t}}$

Yeh ratio hi $R^{2}$ hai: $R^{2} = 1 - \frac{S S _{r es}}{S S _{t o t}}$

YEH INTERPRETATION KYUN KAAM KARTI HAI: Agar $S S_{r es} = 0$ (perfect fit), toh $R^{2} = 1$ . Agar $S S_{r es} = S S_{t o t}$ (model mean se behtar nahi), toh $R^{2} = 0$ .

Figure — R-squared and adjusted R-squared

Simple linear regression ( $y = β_{0} + β_{1} x$ ) ke liye: $R^{2} = r^{2}$ jahan $r$ , $x$ aur $y$ ke beech Pearson correlation coefficient hai.

KYUN? Simple regression mein, explain ki gayi variance ka proportion squared correlation ke barabar hota hai. Yeh multiple regression mein hold nahi karta!

What is Adjusted R-squared?

Alternative form: $R_{a d j}^{2} = 1 - (1 - R^{2}) \frac{n - 1}{n - p - 1}$

Derivation: Why the Penalty?

PROBLEM KYA HAI: Regular $R^{2}$ hamesha badhta hai jab aap zyada features add karte ho, chahe woh features pure noise hi kyun na ho!

YEH KYUN HOTA HAI: Zyada parameters → model training noise ko fit kar leta hai → $S S_{r es}$ ghatta hai → $R^{2}$ badhta hai, chahe model generalize bura kare.

FIX KYA HAI: Degrees of freedom use hone par penalize karo.

Variance ratio se shuru karo:

Residual variance estimate: $\overset{σ}{^}_{r es}^{2} = \frac{S S _{r es}}{n - p - 1}$ (unbiased estimator)
Total variance estimate: $\overset{σ}{^}_{t o t}^{2} = \frac{S S _{t o t}}{n - 1}$ (sample variance)

Adjusted metric banao: $R_{a d j}^{2} = 1 - \frac{σ ^ _{r es}^{2}}{σ ^ _{t o t}^{2}} = 1 - \frac{S S _{r es} / ( n - p - 1 )}{S S _{t o t} / ( n - 1 )}$

$R^{2}$ se relate karo: $R_{a d j}^{2} = 1 - \frac{S S _{r es}}{S S _{t o t}} \cdot \frac{n - 1}{n - p - 1}$ $R_{a d j}^{2} = 1 - (1 - R^{2}) \frac{n - 1}{n - p - 1}$

YEH KYUN KAAM KARTA HAI: Penalty factor $\frac{n - 1}{n - p - 1}$ badhta hai jab $p$ badhta hai. Ek bekar feature add karne se $p$ toh badhta hai lekin $S S_{r es}$ utna nahi ghatta ki compensate ho sake → $R_{a d j}^{2}$ ghatta hai.

Step 1: Mean compute karo $\overset{y}{ˉ} = \frac{200 + 250 + 300 + 350 + 400}{5} = 300$ YEH STEP KYUN? Mean hamara baseline hai—agar hum har baar average predict karein toh kitna achha karenge?

Step 2: $S S_{t o t}$ compute karo $S S_{t o t} = (200 - 300)^{2} + (250 - 300)^{2} + (300 - 300)^{2} + (350 - 300)^{2} + (400 - 300)^{2}$ $= 10000 + 2500 + 0 + 2500 + 10000 = 25000$ YEH STEP KYUN? Yeh total variance hai jo hum explain karne ki koshish kar rahe hain.

Step 3: $S S_{r es}$ compute karo $S S_{r es} = (200 - 210)^{2} + (250 - 240)^{2} + (300 - 310)^{2} + (350 - 340)^{2} + (400 - 390)^{2}$ $= 100 + 100 + 100 + 100 + 100 = 500$ YEH STEP KYUN? Yeh woh variance hai jo hamara model explain karne mein fail raha hai—bacha hua error.

Step 4: $R^{2}$ compute karo $R^{2} = 1 - \frac{500}{25000} = 1 - 0.02 = 0.98$

Interpretation: Model house prices mein 98% variance explain karta hai. Excellent fit!

Step 1: Formula apply karo $R_{a d j}^{2} = 1 - (1 - 0.98) \frac{5 - 1}{5 - 2 - 1}$ YEH STEP KYUN? Humne jo 2 features use kiye hain unke liye penalize karna hai.

Step 2: Calculate karo $R_{a d j}^{2} = 1 - (0.02) \frac{4}{2} = 1 - 0.04 = 0.96$

Interpretation: 2 features ke liye penalize karne ke baad bhi hum 96% variance explain kar rahe hain.

Ab ek bekar 3rd feature daalo (random noise):

New $R^{2} = 0.985$ (overfitting ki wajah se thoda zyada)
$p = 3$

$R_{a d j}^{2} = 1 - (1 - 0.985) \frac{4}{5 - 3 - 1} = 1 - (0.015) \frac{4}{1} = 1 - 0.06 = 0.94$

YEH KYUN GHATTA: Naye feature ne negligible explanatory power add ki lekin ek degree of freedom kha li. $R_{a d j}^{2}$ ne yeh pakad liya!

$R_{a d j}^{2} = 1 - (1 - 0.99) \frac{49}{50 - 48 - 1} = 1 - (0.01) \times 49 = 1 - 0.49 = 0.51$

YEH ITNA KYUN GIRA? $p = 48$ aur $n = 50$ ke saath sirf 1 degree of freedom bachti hai! Model signal nahi, noise fit kar raha hai. $R_{a d j}^{2}$ ne yeh disaster reveal kar diya.

Yeh sahi kyun lagta hai: High $R^{2}$ matlab model training data ko achhi tarah fit karta hai.

Yeh galat kyun hai:

$R^{2}$ training fit measure karta hai, generalization nahi
Overfitted models ka $R^{2}$ high hota hai lekin naye data par fail ho jaate hain
$R^{2}$ detect nahi karta ki tum noise fit kar rahe ho

Fix: Hamesha held-out test data par validate karo. Cross-validation use karo. Feature proliferation ke liye $R_{a d j}^{2}$ check karo.

Yeh sahi kyun lagta hai: $R^{2}$ badhaa, jo improvement jaisa lagta hai.

Yeh galat kyun hai: $R^{2}$ features add karne par kabhi decrease nahi ho sakta (mathematically impossible on training data). Yeh increase pure overfitting ho sakti hai.

Fix: $R_{a d j}^{2}$ use karo. Agar woh ghatta hai ya barely change hota hai, toh naye features bekar hain. Aur behtar: cross-validated metrics use karo.

Yeh sahi kyun lagta hai: Typical linear regression mein intercept ke saath, $R^{2} \geq 0$ hold karta hai.

Yeh galat kyun hai: Agar aap intercept ke bina model fit karo ya itna bura model use karo ki $S S_{r es} > S S_{t o t}$ ho jaaye, toh $R^{2}$ negative ho sakta hai!

Example: Regression ko origin se force karo jab data ka large intercept ho → bekar predictions → negative $R^{2}$ .

Fix: Hamesha intercept include karo jab tak strong theoretical reasons na ho. Agar $R^{2} < 0$ , toh tera model mean predict karne se bhi bura hai.

Key Properties

Property 1: R-squared and Model Complexity

$R^{2}$ features ki sankhya $p$ mein non-decreasing hota hai
$R_{a d j}^{2}$ decrease ho sakta hai agar add kiya gaya feature weak ho
Nested models ke liye (Model A ⊂ Model B), $R_{B}^{2} \geq R_{A}^{2}$ lekin $R_{a d j, B}^{2}$ shayad $< R_{a d j, A}^{2}$ ho

Property 2: Perfect Fit vs. No Fit

$R^{2} = 1 ⟺ S S_{r es} = 0 ⟺ perfect predictions$ $R^{2} = 0 ⟺ S S_{r es} = S S_{t o t} ⟺ model = mean$

Property 3: Relationship Between Metrics

$R_{a d j}^{2} = R^{2} - (1 - R^{2}) \frac{p}{n - p - 1}$ "Penalty" hai $(1 - R^{2}) \frac{p}{n - p - 1}$ , jo badhti hai:

Zyada features ( $p$ ↑)
Chota sample ( $n$ ↓)
Bura fit ( $R^{2}$ ↓)

Recall Ek 12-saal ke bachche ko samjhao

Socho tum andaza laga rahe ho ki tumhare classmates kitne tall hain. Agar tum har baar "average height" guess karo, toh kaafi galat hoge. Ab, agar tum clues use karo jaise unki age aur shoe size? Agar tumhare guesses bahut behtar ho jaayein, toh R-squared measure karta hai KI KITNA BEHTAR.

Isse ek test score ki tarah socho: agar R-squared 0.80 hai, tumhare clues ne tumhe "random guessing" se "perfect prediction" tak ke 80% raaste pe pahuncha diya. R-squared 1.0 matlab 100%—tum ek mind reader ho!

Lekin yahan ek trick hai: kya hoga agar tum ek bekar clue add karo jaise "favorite color"? Yeh actually help nahi karega, lekin shayad luck se tumhara score thoda sa chadh jaaye. Adjusted R-squared ek strict teacher ki tarah hai jo kehta hai "Nope, main bekar clues use karne ke points kaatne wala hoon!" Yeh tumhe honest rakhta hai aur ensure karta hai ki tum sirf wahi clues use karo jo sach mein help karte hain.

When to Use Which

Metric	Kab Use Karein
$R^{2}$	Models compare karte waqt jab same number of features ho
$R_{a d j}^{2}$	Models compare karte waqt jab alag number of features ho
Dono Nahi	Final model evaluation ke liye (test set MSE, MAE, cross-validation use karo)

KYUN? $R^{2}$ aur $R_{a d j}^{2}$ dono training metrics hain. Yeh nahi batate ki model generalize kitna achhi tarah karta hai. Production decisions ke liye, hamesha unseen data par validate karo.

Yaad rakho: Regular R² uss student jaisa hai jo "Maine padha!" count karta hai chahe usne sirf kitaab ko ghoorte hua waqt bitaya ho. Adjusted R² actual exam score hai.

Connections

Linear Regression - squared linear model fit evaluate karta hai
Mean Squared Error - $R^{2}$ MSE se relate karta hai: $R^{2} = 1 - \frac{M S E}{V a r ( y )}$
Overfitting - High $R^{2}$ with low $R_{a d j}^{2}$ overfitting signal karta hai
Feature Selection - $R_{a d j}^{2}$ use karo decide karne ke liye ki features help karte hain ya nahi
Cross-Validation - True performance estimation ke liye $R^{2}$ se behtar hai
Bias-Variance Tradeoff - $R_{a d j}^{2}$ implicitly bias aur variance ko tradeoff karta hai
Model Selection - AIC/BIC alternative penalized fit metrics provide karte hain

#flashcards/ai-ml

R-squared kya measure karta hai? :: Model dwara dependent variable mein explain ki gayi variance ka proportion, jo 0 (koi explanation nahi) se 1 (perfect explanation) tak hota hai.

Sum of squares ke terms mein R-squared ka formula likho.

R^{2} = 1 - \frac{S S _{r es}}{S S _{t o t}}

jahan

S S_{r es}

residual sum of squares hai aur

S S_{t o t}

total sum of squares hai.

R-squared aur adjusted R-squared mein key difference kya hai?

Adjusted R-squared degrees of freedom use karke features ki sankhya ke liye penalize karta hai, jabki regular R-squared features add karne par hamesha badhta hai.

Adjusted R-squared ka formula likho.

R_{a d j}^{2} = 1 - (1 - R^{2}) \frac{n - 1}{n - p - 1}

jahan

n

sample size hai aur

p

predictors ki sankhya hai.

Regular R-squared misleading kyun ho sakta hai?

Yeh features add karne par hamesha badhta hai, chahe woh features pure noise hi kyun na ho, isliye yeh overfitting ko encourage karne ka risk rakhta hai.

Adjusted R-squared R-squared par kab prefer kiya jaata hai?

Jab alag number of features wale models compare karne hों, kyunki yeh model complexity ko account karta hai.

S S_{t o t}

kya represent karta hai?

Total sum of squares:

S S_{t o t} = \sum (y_{i} - \overset{y}{ˉ})^{2}

, dependent variable mein total variance.

S S_{r es}

kya represent karta hai?

Residual sum of squares:

S S_{r es} = \sum (y_{i} - \overset{y}{^}_{i})^{2}

, unexplained variance (prediction errors).

Kya R-squared negative ho sakta hai?

Haan, agar model mean predict karne se bhi bura fit kare (typically jab intercept ke bina fit karo ya bahut bura model use karo).

Adjusted R-squared ki range kya hai?

Yeh negative ho sakta hai (terrible models ke liye) 1 tak, aur hamesha R-squared se ≤ hota hai.

Simple linear regression mein R-squared ka correlation se kya relation hai?

Simple linear regression mein,

R^{2} = r^{2}

jahan

r

Pearson correlation coefficient hai.

R-squared = 0 ka matlab kya hai?

Model koi bhi variance explain nahi karta—yeh hamesha mean predict karne se behtar perform nahi karta.

R-squared = 1 ka matlab kya hai?

Model saare observations ko zero residual error ke saath perfectly predict karta hai.

Adjusted R-squared features add karne par penalize kyun karta hai?

Kyunki har feature ek degree of freedom consume karta hai; agar feature error ko sufficiently reduce nahi karta, toh penalty benefit se zyada ho jaati hai.

Adjusted R-squared mein penalty term kya hai?

(1 - R^{2}) \frac{p}{n - p - 1}

, jo zyada features

p

, chote sample

n

, ya bure fit (lower

R^{2}

) ke saath badhta hai.