R-squared and adjusted R-squared

What is R-squared?

Range: $R^2 \in [0, 1]$ for typical regression (can be negative for bad models without intercept)

Derivation from First Principles

WHY do we need this? The mean $\bar{y}$ is our baseline predictor—if we know nothing, we'd predict the mean. A regression model tries to beat this baseline. We need a metric that compares how much variance remains (residuals) vs. the original variance.

Start with variance decomposition: $SS_{tot} = SS_{reg} + SS_{res}$

WHY this works: Total variance = Explained variance + Unexplained variance

1. Total variance $SS_{tot}$: How much $y$ values spread around their mean $SS_{tot} = \sum_{i=1}^{n}(y_i - \bar{y})^2$

2. Residual variance $SS_{res}$: How much predictions miss the actual values $SS_{res} = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2$

3. Explained variance $SS_{reg}$: How much the model predictions vary from the mean $SS_{reg} = \sum_{i=1}^{n}(\hat{y}_i - \bar{y})^2$

Now derive $R^2$: $SS_{tot} = SS_{reg} + SS_{res}$ $\frac{SS_{tot}}{SS_{tot}} = \frac{SS_{reg}}{SS_{tot}} + \frac{SS_{res}}{SS_{tot}}$ $1 = \frac{SS_{reg}}{SS_{tot}} + \frac{SS_{res}}{SS_{tot}}$ $\frac{SS_{reg}}{SS_{tot}} = 1 - \frac{SS_{res}}{SS_{tot}}$

This ratio is $R^2$: $R^2 = 1 - \frac{SS_{res}}{SS_{tot}}$

WHY this interpretation works: If $SS_{res} = 0$ (perfect fit), $R^2 = 1$. If $SS_{res} = SS_{tot}$ (model no better than mean), $R^2 = 0$.

For simple linear regression ($y = \beta_0 + \beta_1 x$): $R^2 = r^2$ where $r$ is the Pearson correlation coefficient between $x$ and $y$.

WHY? In simple regression, the proportion of variance explained equals the squared correlation. This doesn't hold for multiple regression!

What is Adjusted R-squared?

Alternative form: $R^2_{adj} = 1 - (1-R^2)\frac{n-1}{n-p-1}$

Derivation: Why the Penalty?

THE PROBLEM: Regular $R^2$ always increases when you add more features, even if those features are pure noise!

WHY this happens: More parameters → model can fit training noise → $SS_{res}$ decreases → $R^2$ increases, even if the model generalizes worse.

THE FIX: Penalize by degrees of freedom used.

Start with the variance ratio:

• Residual variance estimate: $\hat{\sigma}^2_{res} = \frac{SS_{res}}{n-p-1}$ (unbiased estimator)
• Total variance estimate: $\hat{\sigma}^2_{tot} = \frac{SS_{tot}}{n-1}$ (sample variance)

Build the adjusted metric: $R^2_{adj} = 1 - \frac{\hat{\sigma}^2_{res}}{\hat{\sigma}^2_{tot}} = 1 - \frac{SS_{res}/(n-p-1)}{SS_{tot}/(n-1)}$

Relate to $R^2$: $R^2_{adj} = 1 - \frac{SS_{res}}{SS_{tot}} \cdot \frac{n-1}{n-p-1}$ $R^2_{adj} = 1 - (1-R^2)\frac{n-1}{n-p-1}$

WHY this works: The penalty factor $\frac{n-1}{n-p-1}$ increases as $p$ increases. Adding a useless feature increases $p$ but doesn't decrease $SS_{res}$ enough to compensate → $R^2_{adj}$ decreases.

Step 1: Compute mean $\bar{y} = \frac{200+250+300+350+400}{5} = 300$ WHY this step? The mean is our baseline—how well would we do predicting the average every time?

Step 2: Compute $SS_{tot}$ $SS_{tot} = (200-300)^2 + (250-300)^2 + (300-300)^2 + (350-300)^2 + (400-300)^2$ $= 10000 + 2500 + 0 + 2500 + 10000 = 25000$ WHY this step? This is the total variance we're trying to explain.

Step 3: Compute $SS_{res}$ $SS_{res} = (200-210)^2 + (250-240)^2 + (300-310)^2 + (350-340)^2 + (400-390)^2$ $= 100 + 100 + 100 + 100 + 100 = 500$ WHY this step? This is the variance our model fails to explain—the leftover error.

Step 4: Compute $R^2$ $R^2 = 1 - \frac{500}{25000} = 1 - 0.02 = 0.98$

Interpretation: The model explains 98% of variance in house prices. Excellent fit!

Step 1: Apply the formula $R^2_{adj} = 1 - (1-0.98)\frac{5-1}{5-2-1}$ WHY this step? We need to penalize for the 2 features we used.

Step 2: Calculate $R^2_{adj} = 1 - (0.02)\frac{4}{2} = 1 - 0.04 = 0.96$

Interpretation: After penalizing for 2 features, we still explain 96% of variance.

Now add useless 3rd feature (random noise):

• New $R^2 = 0.985$ (slightly higher due to overfitting)
• $p = 3$

$R^2_{adj} = 1 - (1-0.985)\frac{4}{5-3-1} = 1 - (0.015)\frac{4}{1} = 1 - 0.06 = 0.94$

WHY it decreased: The new feature added negligible explanatory power but consumed a degree of freedom. $R^2_{adj}$ caught this!

$R^2_{adj} = 1 - (1-0.99)\frac{49}{50-48-1} = 1 - (0.01)\times 49 = 1 - 0.49 = 0.51$

WHY the huge drop? With $p=48$ and $n=50$, you only have 1 degree of freedom left! The model is fitting noise, not signal. $R^2_{adj}$ reveals this disaster.

Why it feels right: High $R^2$ means the model fits the training data well.

Why it's wrong:

1. $R^2$ measures training fit, not generalization
2. Overfitted models have high $R^2$ but fail on new data
3. $R^2$ doesn't detect if you're fitting noise

The fix: Always validate on held-out test data. Use cross-validation. Check $R^2_{adj}$ for feature proliferation.

Why it feels right: $R^2$ increased, which sounds like improvement.

Why it's wrong: $R^2$ cannot decrease when adding features (mathematically impossible on training data). The increase might be pure overfitting.

The fix: Use $R^2_{adj}$. If it decreases or barely changes, the new features are useless. Better yet, use cross-validated metrics.

Why it feels right: In typical linear regression with intercept, $R^2 \geq 0$ holds.

Why it's wrong: If you fit a model without an intercept or use a model so bad that $SS_{res} > SS_{tot}$, $R^2$ can be negative!

Example: Force regression through origin when data has large intercept → terrible predictions → negative $R^2$.

The fix: Always include an intercept unless you have strong theoretical reasons. If $R^2 < 0$, your model is worse than predicting the mean.

Key Properties

Property 1: R-squared and Model Complexity

• $R^2$ is non-decreasing in $p$ (number of features)
• $R^2_{adj}$ can decrease if added feature is weak
• For nested models (Model A ⊂ Model B), $R^2_B \geq R^2_A$ but $R^2_{adj,B}$ may be $< R^2_{adj,A}$

Property 2: Perfect Fit vs. No Fit

$R^2 = 1 \iff SS_{res} = 0 \iff \text{perfect predictions}$ $R^2 = 0 \iff SS_{res} = SS_{tot} \iff \text{model = mean}$

Property 3: Relationship Between Metrics

$R^2_{adj} = R^2 - (1-R^2)\frac{p}{n-p-1}$ The "penalty" is $(1-R^2)\frac{p}{n-p-1}$, which grows with:

• More features ($p$↑)
• Smaller sample ($n$ ↓)
• Worse fit ($R^2$ ↓)

Think of it like a test score: if R-squared is 0.80, your clues helped you get 80% of the way from "random guessing" to "perfect prediction." An R-squared of 1.0 is 100%—you're a mind reader!

But here's the trick: what if you add a silly clue like "favorite color"? It won't actually help, but you might get lucky and your score goes up a tiny bit just by chance. Adjusted R-squared is like a strict teacher who says "Nope, I'm taking points off for using useless clues!" It keeps you honest and makes sure you only use clues that truly help.

When to Use Which

WHY? Both $R^2$ and $R^2_{adj}$ are training metrics. They don't tell you how well the model generalizes. For production decisions, always validate on unseen data.

Remember: Regular R² is like a student who counts "I studied!" even if they just stared at the book. Adjusted R² is the actual exam score.

Connections

• Linear Regression -squared evaluates linear model fit
• Mean Squared Error - $R^2$ relates to MSE: $R^2 = 1 - \frac{MSE}{Var(y)}$
• Overfitting - High $R^2$ with low $R^2_{adj}$ signals overfitting
• Feature Selection - Use $R^2_{adj}$ to decide if features help
• Cross-Validation - Better than $R^2$ for true performance estimation
• Bias-Variance Tradeoff - $R^2_{adj}$ implicitly trades off bias and variance
• Model Selection - AIC/BIC provide alternative penalized fit metrics

#flashcards/ai-ml

What does R-squared measure? :: The proportion of variance in the dependent variable explained by the model, ranging from 0 (no explanation) to 1 (perfect explanation).

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)