R-squared and adjusted R-squared: Exercises — AI-ML deep dive

Before we start, let me re-anchor the three quantities so nothing below uses a symbol you haven't seen defined on this page.

Figure — R-squared and adjusted R-squared

Look at the figure: the red vertical stubs are the residuals $(y_{i} - \overset{y}{^}_{i})$ that build $S S_{r es}$ ; the grey stubs to the flat line build $S S_{t o t}$ . Every exercise below is really about the size of those red stubs — the smaller they are compared to the grey stubs, the closer $R^{2}$ climbs to $1$ .

Intuition Where the penalty factor comes from (the WHY behind

$\frac{n - 1}{n - p - 1}$ ) The fee $\frac{n - 1}{n - p - 1}$ is not arbitrary — it is a ratio of degrees of freedom, which just means "how many independent facts are left after you spend some fitting the model."

$S S_{t o t}$ is built after spending one fact to compute $\overset{y}{ˉ}$ , so it has $n - 1$ free numbers left.
$S S_{r es}$ is built after spending $p + 1$ facts (the intercept plus $p$ feature-weights), so it has $n - p - 1$ free numbers left.

Look at the second figure: on the left, few features leave many free numbers ( $n - p - 1$ large), so the fee $\frac{n - 1}{n - p - 1} \approx 1$ and $R_{a d j}^{2} \approx R^{2}$ . On the right, features gobble up the free numbers ( $n - p - 1$ shrinks toward $0$ ), so the fee balloons and $R_{a d j}^{2}$ crashes. The formula literally measures "error per remaining free number" against "spread per remaining free number" — a fair, size-matched comparison instead of raw sums.

Level 1 — Recognition

L1.1 — Read the formula

State, in one sentence each, what $S S_{r es}$ , $S S_{t o t}$ , and $R^{2}$ measure. Then: if a model predicts the mean $\overset{y}{ˉ}$ for every point, what is $R^{2}$ ?

Recall Solution

$S S_{r es}$ = total squared distance between actual points and the model's predictions (leftover error).
$S S_{t o t}$ = total squared distance between actual points and their average (the whole spread).
$R^{2}$ = the fraction of that spread the model removed.

If the model predicts $\overset{y}{ˉ}$ everywhere, then $\overset{y}{^}_{i} = \overset{y}{ˉ}$ , so $S S_{r es} = \sum (y_{i} - \overset{y}{ˉ})^{2} = S S_{t o t}$ . Then $R^{2} = 1 - \frac{S S _{t o t}}{S S _{t o t}} = 1 - 1 = 0.$ A useless model scores exactly $0$ — this is the baseline the whole idea is built on. In the first figure, this is the case where every red stub matches its grey stub exactly.

L1.2 — Plug and chug

Given $S S_{r es} = 300$ and $S S_{t o t} = 1500$ , compute $R^{2}$ .

Recall Solution

$R^{2} = 1 - \frac{300}{1500} = 1 - 0.2 = 0.8.$ The model explains $80%$ of the spread.

L1.3 — Direction of the penalty

Without computing anything, decide: as you add more useless features $p$ (with $n$ fixed and $R^{2}$ unchanged), does $R_{a d j}^{2}$ go up or down? Say why using the formula $R_{a d j}^{2} = 1 - (1 - R^{2}) \frac{n - 1}{n - p - 1}$ .

Recall Solution

Increasing $p$ shrinks the denominator $n - p - 1$ , so the fraction $\frac{n - 1}{n - p - 1}$ grows (fewer free numbers left — see the second figure's right side). That makes $(1 - R^{2}) \times (bigger number)$ larger, and we subtract it, so $R_{a d j}^{2}$ goes down. Useless features are punished even when $R^{2}$ stays flat.

Level 2 — Application

L2.1 — Full $R^{2}$ from raw data

Actuals $y = [10, 12, 14, 16, 18]$ , predictions $\overset{y}{^} = [11, 11, 15, 15, 19]$ . Compute $\overset{y}{ˉ}$ , $S S_{t o t}$ , $S S_{r es}$ , and $R^{2}$ .

Recall Solution

$\overset{y}{ˉ} = \frac{10 + 12 + 14 + 16 + 18}{5} = \frac{70}{5} = 14.$ $S S_{t o t} = (- 4)^{2} + (- 2)^{2} + 0^{2} + 2^{2} + 4^{2} = 16 + 4 + 0 + 4 + 16 = 40.$ Residuals $y_{i} - \overset{y}{^}_{i} = [- 1, 1, - 1, 1, - 1]$ (these are exactly the red stubs in the first figure): $S S_{r es} = 1 + 1 + 1 + 1 + 1 = 5.$ $R^{2} = 1 - \frac{5}{40} = 1 - 0.125 = 0.875.$

L2.2 — Adjusted $R^{2}$

Using $R^{2} = 0.875$ from L2.1, with $n = 5$ and $p = 2$ features, compute $R_{a d j}^{2}$ .

Recall Solution

$R_{a d j}^{2} = 1 - (1 - 0.875) \frac{5 - 1}{5 - 2 - 1} = 1 - 0.125 \cdot \frac{4}{2} = 1 - 0.125 \cdot 2 = 1 - 0.25 = 0.75.$ The two features cost us $0.875 \to 0.75$ . Note the free numbers left: $n - p - 1 = 2$ , so the fee $\frac{4}{2} = 2$ is already sizeable.

L2.3 — Back-solve for $S S_{r es}$

A model reports $R^{2} = 0.6$ on data with $S S_{t o t} = 2000$ . What is $S S_{r es}$ ?

Recall Solution

$R^{2} = 1 - \frac{S S _{r es}}{S S _{t o t}} \Rightarrow \frac{S S _{r es}}{S S _{t o t}} = 1 - R^{2} = 0.4.$ $S S_{r es} = 0.4 \times 2000 = 800.$

Level 3 — Analysis

L3.1 — Does the new feature earn its keep?

Model A: $R^{2} = 0.80$ , $p = 3$ . You add one feature, giving Model B: $R^{2} = 0.82$ , $p = 4$ . Both use $n = 30$ . Compute $R_{a d j}^{2}$ for each and decide which model to keep.

Recall Solution

Model A: $R_{a d j, A}^{2} = 1 - (1 - 0.80) \frac{29}{30 - 3 - 1} = 1 - 0.20 \cdot \frac{29}{26} = 1 - 0.20 \cdot 1.11538 \dots = 1 - 0.223077 = 0.776923.$ Model B: $R_{a d j, B}^{2} = 1 - (1 - 0.82) \frac{29}{30 - 4 - 1} = 1 - 0.18 \cdot \frac{29}{25} = 1 - 0.18 \cdot 1.16 = 1 - 0.2088 = 0.7912.$ $R_{a d j, B}^{2} (0.7912) > R_{a d j, A}^{2} (0.7769)$ , so the new feature earned its place — keep Model B. The $R^{2}$ rise from $0.80$ to $0.82$ was big enough to beat the extra-parameter fee.

L3.2 — The overfitting cliff

$n = 20$ , and a model achieves $R^{2} = 0.95$ using $p = 18$ features. Compute $R_{a d j}^{2}$ . What does it reveal?

Recall Solution

$R_{a d j}^{2} = 1 - (1 - 0.95) \frac{20 - 1}{20 - 18 - 1} = 1 - 0.05 \cdot \frac{19}{1} = 1 - 0.95 = 0.05.$ A dazzling $R^{2} = 0.95$ collapses to $R_{a d j}^{2} = 0.05$ . With only $20 - 18 - 1 = 1$ residual degree of freedom, the model is memorising noise, not learning signal — the adjusted score has stripped away almost all of the apparent fit. This is textbook Overfitting — the adjusted score refuses to be fooled.

L3.3 — Negative $R^{2}$

A no-intercept model (see the "What intercept means" box above) on data with $S S_{t o t} = 500$ produces predictions with $S S_{r es} = 650$ . Compute $R^{2}$ and interpret.

Recall Solution

$R^{2} = 1 - \frac{650}{500} = 1 - 1.3 = - 0.3.$ $R^{2}$ is negative — the model is worse than just predicting the mean (which would score $0$ ). Because the intercept $b_{0}$ was forced to $0$ , the line cannot match the data's average level, so $S S_{r es}$ (error from predictions) exceeds $S S_{t o t}$ (error from the flat mean line). Not a bug; a verdict.

Level 4 — Synthesis

L4.1 — Feature-selection decision table

You have $n = 100$ points and three nested models. Compute each $R_{a d j}^{2}$ and recommend one.

Model	$R^{2}$	$p$
M1	0.70	2
M2	0.75	5
M3	0.76	12

Recall Solution

M1: $R_{a d j}^{2} = 1 - (1 - 0.70) \frac{99}{100 - 2 - 1} = 1 - 0.30 \cdot \frac{99}{97} = 1 - 0.30 \cdot 1.020619 = 1 - 0.306186 = 0.693814.$ M2: $R_{a d j}^{2} = 1 - (1 - 0.75) \frac{99}{100 - 5 - 1} = 1 - 0.25 \cdot \frac{99}{94} = 1 - 0.25 \cdot 1.053191 = 1 - 0.263298 = 0.736702.$ M3: $R_{a d j}^{2} = 1 - (1 - 0.76) \frac{99}{100 - 12 - 1} = 1 - 0.24 \cdot \frac{99}{87} = 1 - 0.24 \cdot 1.137931 = 1 - 0.273103 = 0.726897.$

Ranking: M2 ( $0.7367$ ) > M3 ( $0.7269$ ) > M1 ( $0.6938$ ). M3 has the highest raw $R^{2}$ but its 7 extra features over M2 buy only $0.01$ more explained variance — not worth the degrees of freedom. Recommend M2. This is exactly what Feature Selection and Model Selection aim to formalise.

L4.2 — Derive the penalty gap

Show algebraically that $R_{a d j}^{2} = R^{2} - (1 - R^{2}) \frac{p}{n - p - 1}$ , starting from $R_{a d j}^{2} = 1 - (1 - R^{2}) \frac{n - 1}{n - p - 1}$ . Then verify with $R^{2} = 0.80$ , $n = 30$ , $p = 3$ (matches L3.1's Model A).

Recall Solution

Start: $R_{a d j}^{2} = 1 - (1 - R^{2}) \frac{n - 1}{n - p - 1} .$ Write $n - 1 = (n - p - 1) + p$ : $\frac{n - 1}{n - p - 1} = \frac{( n - p - 1 ) + p}{n - p - 1} = 1 + \frac{p}{n - p - 1} .$ Substitute: $R_{a d j}^{2} = 1 - (1 - R^{2}) (1 + \frac{p}{n - p - 1}) = 1 - (1 - R^{2}) - (1 - R^{2}) \frac{p}{n - p - 1} .$ Since $1 - (1 - R^{2}) = R^{2}$ : $R_{a d j}^{2} = R^{2} - (1 - R^{2}) \frac{p}{n - p - 1} .$ Check ( $R^{2} = 0.80, n = 30, p = 3$ ): $0.80 - 0.20 \cdot \frac{3}{26} = 0.80 - 0.20 \cdot 0.115385 = 0.80 - 0.023077 = 0.776923$ — matches L3.1's $R_{a d j, A}^{2}$ exactly. The subtracted term is literally the "gap" between $R^{2}$ and $R_{a d j}^{2}$ .

Level 5 — Mastery

L5.1 — End-to-end from data to decision

Raw data, two competing fits on the same $n = 6$ points. $y = [3, 5, 7, 9, 11, 13]$ . Model A (1 feature) predicts $\overset{y}{^}_{A} = [3, 5, 7, 9, 11, 13]$ — a perfect line. Model B (3 features) predicts $\overset{y}{^}_{B} = [3.5, 4.5, 7, 9, 10.5, 13.5]$ .

(a) Compute $R^{2}$ and $R_{a d j}^{2}$ for both. (b) Which model wins, and what deeper lesson does the pair teach?

Recall Solution

Mean: $\overset{y}{ˉ} = \frac{3 + 5 + 7 + 9 + 11 + 13}{6} = \frac{48}{6} = 8.$ $S S_{t o t} = (- 5)^{2} + (- 3)^{2} + (- 1)^{2} + 1^{2} + 3^{2} + 5^{2} = 25 + 9 + 1 + 1 + 9 + 25 = 70.$

Model A — residuals all $0$ : $S S_{r es, A} = 0, R_{A}^{2} = 1 - \frac{0}{70} = 1.0.$ $R_{a d j, A}^{2} = 1 - (1 - 1) \frac{6 - 1}{6 - 1 - 1} = 1 - 0 = 1.0.$

Model B — residuals $y_{i} - \overset{y}{^}_{B, i} = [- 0.5, 0.5, 0, 0, 0.5, - 0.5]$ : $S S_{r es, B} = 0.25 + 0.25 + 0 + 0 + 0.25 + 0.25 = 1.0.$ $R_{B}^{2} = 1 - \frac{1.0}{70} = 1 - 0.0142857 = 0.985714.$ $R_{a d j, B}^{2} = 1 - (1 - 0.985714) \frac{6 - 1}{6 - 3 - 1} = 1 - 0.0142857 \cdot \frac{5}{2} = 1 - 0.0142857 \cdot 2.5 = 1 - 0.0357143 = 0.964286.$

(b) Winner: Model A on every metric ( $R^{2} = 1$ , $R_{a d j}^{2} = 1$ ). The lesson: Model B used 3 features to do a worse job than Model A's 1 feature. More parameters bought nothing — in fact they slightly increased error. $R_{a d j}^{2}$ makes the waste explicit ( $0.9643$ vs $1.0$ ). Simpler wins when it fits; this is the whole spirit of Model Selection and guards against Overfitting. (See Linear Regression for why a straight generating rule like $y = 2 x + 1$ is captured exactly by one feature, and Mean Squared Error for the per-point view of these same residuals.)

L5.2 — Sensitivity: how many features break the model?

With $n = 25$ and a genuine $R^{2} = 0.90$ that stays fixed, find the largest integer $p$ for which $R_{a d j}^{2}$ is still positive. (Assume adding features keeps $R^{2}$ at $0.90$ — the pessimistic "useless features" case.)

Recall Solution

$R_{a d j}^{2} = 1 - (1 - 0.90) \frac{24}{25 - p - 1} = 1 - 0.10 \cdot \frac{24}{24 - p} = 1 - \frac{2.4}{24 - p} .$ Positivity requires $\frac{2.4}{24 - p} < 1$ , i.e. $24 - p > 2.4$ , i.e. $p < 21.6$ . Since $p$ is an integer and we need $24 - p > 0$ (positive residual df), the largest is $p = 21.$ Check $p = 21$ : $R_{a d j}^{2} = 1 - \frac{2.4}{3} = 1 - 0.8 = 0.2 > 0.$ ✔ Check $p = 22$ : $R_{a d j}^{2} = 1 - \frac{2.4}{2} = 1 - 1.2 = - 0.2 < 0.$ ✘ So even a strong $R^{2} = 0.90$ turns worthless once you cram in $22$ useless features on $25$ points — the degrees of freedom run out.