R-squared and adjusted R-squared: Question bank

Two quick reminders so every symbol below is earned:

$S S_{r es} = \sum_{i} (y_{i} - \overset{y}{^}_{i})^{2}$ — how badly your predictions $\overset{y}{^}_{i}$ miss the true values $y_{i}$ (leftover error).
$S S_{t o t} = \sum_{i} (y_{i} - \overset{y}{ˉ})^{2}$ — how spread out the data is around its mean $\overset{y}{ˉ}$ (the baseline "predict the average" error).
$R^{2} = 1 - \frac{S S _{r es}}{S S _{t o t}}$ , and $R_{a d j}^{2} = 1 - (1 - R^{2}) \frac{n - 1}{n - p - 1}$ with $n$ = data points, $p$ = number of predictors.

True or false — justify

A high

R^{2}

guarantees the model predicts well on new, unseen data.

False.

R^{2}

scores fit on the data the model already saw; a model can memorize training noise, giving high

R^{2}

yet failing on new data — this is Overfitting. Only held-out Cross-Validation tests generalization.

R^{2} = 0

means the model is completely random and useless.

Roughly true in effect but precise reason matters:

R^{2} = 0

means

S S_{r es} = S S_{t o t}

, i.e. the model does exactly as well as always predicting the mean

\overset{y}{ˉ}

— no better, no worse, not "random noise".

R^{2}

can never be negative.

False. If the model is worse than predicting the mean (e.g. regression forced through the origin, or a badly misspecified model), then

S S_{r es} > S S_{t o t}

and

R^{2} = 1 - \frac{S S _{r es}}{S S _{t o t}} < 0

Adding any new feature can never decrease the training

R^{2}

True. The optimizer can always set the new coefficient to 0 and recover the old fit, so

S S_{r es}

can only stay the same or shrink —

R^{2}

is non-decreasing in

p

on training data.

Adding a new feature can never decrease adjusted

R^{2}

False.

R_{a d j}^{2}

carries the penalty factor

\frac{n - 1}{n - p - 1}

that grows with

p

; a weak feature raises

p

without shrinking

S S_{r es}

enough, so

R_{a d j}^{2}

drops. That's the whole point of the adjustment.

For simple linear regression,

R^{2}

equals the square of the Pearson correlation

r

True for one predictor with an intercept:

R^{2} = r^{2}

. This does NOT extend to multiple regression, where

R^{2}

is the squared correlation between

y

and

\overset{y}{^}

, not between

y

and any single

x

R_{a d j}^{2}

is always less than or equal to

R^{2}

True whenever

p \geq 1

, because the penalty factor

\frac{n - 1}{n - p - 1} \geq 1

multiplies the non-negative quantity

(1 - R^{2})

, pulling

R_{a d j}^{2}

down (they're equal only when

p = 0

R^{2} = 1

A model with

R^{2} = 0.98

is always better than one with

R^{2} = 0.90

False. If the two models were fit on different datasets, or the 0.98 model used far more features and overfit, the comparison is meaningless.

R^{2}

only compares models on the same target and data, and even then Overfitting can inflate it.

Spot the error

"My

R^{2}

went from 0.80 to 0.85 after adding 10 features, so the model definitely improved."

The error: training

R^{2}

cannot decrease when features are added, so an increase is expected regardless of whether the features help. Check

R_{a d j}^{2}

or cross-validated error — the rise may be pure Overfitting.

R^{2} = 0.99

with 48 features and 50 data points — an excellent model."

The error: only

n - p - 1 = 1

degree of freedom remains, so the model is fitting noise.

R_{a d j}^{2}

collapses to about 0.51 here, exposing the disaster that raw

R^{2}

hid.

"I removed the intercept to simplify the model, and now

R^{2}

is negative — must be a code bug."

Not a bug. Forcing the line through the origin when the data has a nonzero mean makes predictions worse than the mean baseline, so

S S_{r es} > S S_{t o t}

and

R^{2}

legitimately goes negative.

R^{2}

tells me how much of the variance in

x

my model explains."

The error:

R^{2}

explains variance in the target

y

(via

S S_{t o t} = \sum (y_{i} - \overset{y}{ˉ})^{2}

), not in the inputs

x

. The predictors are the tools, not the thing being explained.

"To pick the best subset of features, I'll just choose whichever gives the highest

R^{2}

The error: highest

R^{2}

always means "use every feature," so this rule never removes anything. Use

R_{a d j}^{2}

, cross-validated score, or proper Feature Selection / Model Selection criteria instead.

R_{a d j}^{2}

can also be interpreted directly as a proportion of variance explained."

The error: unlike

R^{2}

R_{a d j}^{2}

is an adjusted comparison of variance estimates and has no clean "percent explained" meaning — it can even go negative for tiny samples with many predictors.

Why questions

Why do we compare against the mean

\overset{y}{ˉ}

specifically, and not zero?

The mean is the best constant predictor when you know nothing about

x

— it minimizes squared error among all constants. So

S S_{t o t}

is the "informed baseline," making

R^{2}

a fair measure of how much the model beats knowing nothing.

Why does

R^{2}

always increase (or stay flat) as you add features, mathematically?

The least-squares fit minimizes

S S_{r es}

over more parameters; the old solution is still available (set new coefficients to 0), so the new minimum can only be

\leq

the old one. Smaller

S S_{r es}

forces larger

R^{2}

Why divide

S S_{r es}

n - p - 1

instead of just

n

in the adjusted formula?

Each estimated parameter (intercept +

p

slopes) "uses up" a degree of freedom, so

\overset{σ}{^}_{r es}^{2} = \frac{S S _{r es}}{n - p - 1}

is the unbiased variance estimate. Using

n - p - 1

is exactly what injects the complexity penalty.

Why does the penalty factor

\frac{n - 1}{n - p - 1}

get harsher when

n

is small?

With few data points, each added predictor eats a larger fraction of the available degrees of freedom, so

n - p - 1

shrinks fast and the factor balloons — small samples punish extra features much more, discouraging Overfitting.

Why is

R^{2}

alone not a good tool for Feature Selection?

Because it monotonically rewards more features, it can't tell a helpful feature from a noisy one. You need a criterion that trades fit against complexity —

R_{a d j}^{2}

, AIC/BIC, or Cross-Validation error.

Why does a high training

R^{2}

sometimes signal high variance in the Bias-Variance Tradeoff sense?

A very flexible model can drive

S S_{r es}

near zero on training data (high

R^{2}

) by chasing noise, which is exactly the high-variance regime — great train fit, poor test performance.

Why does

R^{2} = r^{2}

hold for simple regression but break for multiple regression?

With one predictor, the fitted line's explained variance equals the squared linear correlation between

x

and

y

. With many predictors there's no single

x

to correlate with;

R^{2}

becomes the squared correlation between

y

and the combined prediction

\overset{y}{^}

Edge cases

What is

R^{2}

when the model predicts perfectly (

\overset{y}{^}_{i} = y_{i}

for all

i

S S_{r es} = 0

, so

R^{2} = 1 - 0 = 1

— the maximum. Every wiggle in

y

is accounted for.

What is

R^{2}

when every prediction equals the mean (

\overset{y}{^}_{i} = \overset{y}{ˉ}

S S_{r es} = S S_{t o t}

, so

R^{2} = 0

. The model has learned nothing beyond the baseline.

What happens to

R_{a d j}^{2}

when

p = 0

(only an intercept, no predictors)?

The factor

\frac{n - 1}{n - p - 1} = \frac{n - 1}{n - 1} = 1

, so

R_{a d j}^{2} = R^{2}

. With no features to penalize, the two metrics coincide.

What happens to the adjusted formula as

p \to n - 1

(features approach data points)?

The denominator

n - p - 1 \to 0

, so the penalty factor blows up toward infinity and

R_{a d j}^{2} \to - \infty

. This is the math screaming that you have too many features for your data.

If two variables are perfectly correlated with the target and you include both, what breaks?

The design becomes (near-)singular — infinitely many coefficient combinations give the same fit, so

R^{2}

is well-defined but individual coefficients are not. This is why collinearity matters even when

R^{2}

looks fine; see Feature Selection.

What does

R^{2}

mean if the target

y

is constant (all

y_{i}

equal)?

Then

S S_{t o t} = 0

and

R^{2} = 1 - \frac{0}{0}

is undefined. There's no variance to explain, so

R^{2}

has no meaning — the metric assumes the target actually varies.

Recall Self-check before moving on

One-line summary of the whole page ::: $R^{2}$ measures training fit and always rewards more features; $R_{a d j}^{2}$ penalizes complexity so it can fall — but neither replaces Cross-Validation for judging real predictive power.