Visual walkthrough — R-squared and adjusted R-squared

---

Step 1 — The data and the one number we start with

WHAT. We have a handful of points. Each point is one observation: a thing we measured (say the true price of a house), written $y_i$. The little $i$ is just a counter — point number 1, point number 2, and so on, up to point number $n$ (here $n=5$).

WHY. Before we can talk about how good a model is, we need something to compare it against. The simplest possible "model" is a lazy one: ignore everything and always guess the average. That average is our zero-effort baseline. If a real model can't beat guessing the average, it is worthless.

The average (we call it the mean) is:

$\bar{y}=\underset{\text{split into }n\text{ equal shares}}{\underbrace{\frac{1}{n}}}\underset{\text{add up every point}}{\underbrace{\sum _{i=1}^n{y}_i}}$

The bar over $y$ means "averaged". The stretched-S symbol $\sum$ (Greek "S", for Sum) just says add these up, starting the counter $i$ at $1$ and stopping at $n$.

PICTURE. The dots are our data. The dashed horizontal line is $\bar{y}$ — the flat, boring baseline model that predicts the same value no matter what.

---

Step 2 — Total spread: how far the dots stray from the baseline

WHAT. For every dot, draw a vertical stick from the dot down to the baseline line $\bar{y}$. The length of that stick is $(y_i-\bar{y})$: how far this point sits above or below average.

WHY. We want ONE number for "how much do these dots vary overall?" We can't just add the sticks — points above average give positive lengths and points below give negative, and they'd cancel to nearly zero. So we square each length first. Squaring does two jobs at once: it kills the minus signs, and it makes big deviations count much more than small ones.

$S{S}_{tot}={\sum }_{i=1}^n\underset{\text{stick length}}{\underbrace{(y_i-\bar{y})}}{}^2$

"$SS$" = Sum of Squares; the subscript "$tot$" = total. So $S{S}_{tot}$ is "the total squared spread of the data around its own mean". This is the size of the job — the whole pile of variation any model is trying to explain.

PICTURE. Each grey stick is a deviation from the baseline. The little squares hanging off them are the squared lengths — literally areas. $S{S}_{tot}$ is the total area.

---

Step 3 — The model, and the errors it leaves behind

WHAT. Now a real model enters. It looks at each point and produces a prediction, written ${\hat{y}}_i$ (the little hat "$\hat{}$" means "estimated / predicted", to distinguish it from the true $y_i$). In simple linear regression this prediction is the sloped line from Linear Regression.

For each point, the gap between what actually happened and what the model guessed is the residual:

${\text{residual}}_i=y_i-{\hat{y}}_i$

WHY. These leftover gaps are exactly what the model failed to explain. If we square and sum them, we get the model's remaining error — the same quantity minimised during training (this is the Mean Squared Error, just without dividing by $n$):

$S{S}_{res}={\sum }_{i=1}^n\underset{\text{residual: true }-\text{ predicted}}{\underbrace{(y_i-{\hat{y}}_i)}}{}^2$

Subscript "$res$" = residual. Small $S{S}_{res}$ means the line hugs the dots; large $S{S}_{res}$ means the line misses badly.

PICTURE. The orange line is the model. The red sticks now run from each dot to the line (not the flat baseline). Those red sticks are the residuals — the errors that survive.

---

Step 4 — Splitting the spread into two clean piles

WHAT. There is a third quantity: how far the model's line sits from the flat baseline. That is the part of the spread the model did account for:

$S{S}_{reg}={\sum }_{i=1}^n\underset{\text{prediction }-\text{ baseline}}{\underbrace{({\hat{y}}_i-\bar{y})}}{}^2$

Subscript "$reg$" = regression (the explained part).

WHY. Here is the beautiful fact. For a least-squares line with an intercept, the total spread splits perfectly into these two piles with no overlap:

$\underset{\text{total spread}}{\underbrace{S{S}_{tot}}}=\underset{\text{explained}}{\underbrace{S{S}_{reg}}}+\underset{\text{unexplained}}{\underbrace{S{S}_{res}}}$

Read it in words: total variation = variation the model captured + variation still left over. The reason there is no cross-term (no leftover mess) is geometric: the least-squares line makes the residual sticks perpendicular, in a high-dimensional sense, to the explained part — like the two short sides of a right triangle whose squares add to the hypotenuse's square.

PICTURE. For one point we split its total stick (dot → baseline) into two segments: dot → line (red, unexplained) and line → baseline (teal, explained). Their squared lengths add up to the total's squared length.

---

Step 5 — Turning piles into a score: $R^2$

WHAT. Divide the whole splitting equation by $S{S}_{tot}$ so everything becomes a fraction of the total:

\quad\Longrightarrow\quad 1 = \underbrace{\frac{SS_{reg}}{SS_{tot}}}_{\text{fraction explained}} + \underbrace{\frac{SS_{res}}{SS_{tot}}}_{\text{fraction left over}}$$ **WHY.** The fraction the model **explains** is the one we care about, so we name it $R^2$. Rearranging the line above gives the two identical faces of the same coin: $$\boxed{\,R^2 \;=\; \frac{SS_{reg}}{SS_{tot}} \;=\; 1 - \frac{SS_{res}}{SS_{tot}}\,}$$ - Left form: "of the whole spread, what share did I capture?" - Right form: "start at a perfect 100\%, subtract the share I wasted as error." Because both piles are non-negative and add to $SS_{tot}$, a good model gives $R^2$ near $1$ and a lazy one near $0$. **PICTURE.** A single stacked bar of total height $SS_{tot}$: the teal chunk (explained) is $R^2$ of the bar, the red chunk (residual) is $1-R^2$. That is *all* $R^2$ is — the height of the teal slice as a fraction. --- ## Step 6 — The edge cases: $R^2 = 1$, $R^2 = 0$, and $R^2 < 0$ **WHAT.** We must show every possible outcome, not just the pretty one. - **Perfect fit** — line passes through every dot. Every residual is zero, so $SS_{res}=0$ and $R^2 = 1-0 = 1$. - **No better than the mean** — the fitted line is flat at $\bar{y}$. Then predictions equal the baseline, $SS_{res}=SS_{tot}$, and $R^2 = 1-1 = 0$. - **Worse than the mean** — this surprises people. If the model is *forced* to be bad (e.g. no intercept, wrong slope), the residual sticks can be **longer** than the baseline sticks, so $SS_{res} > SS_{tot}$ and $R^2$ goes **negative**. Negative $R^2$ is not a bug; it is the model announcing "you'd have done better guessing the average." **WHY.** $R^2 = 1 - \dfrac{SS_{res}}{SS_{tot}}$. The fraction $\frac{SS_{res}}{SS_{tot}}$ is $0$ for a perfect fit, $1$ for the mean-model, and can exceed $1$ for a genuinely bad model — which is exactly the three regimes above. This is why the parent note warns that $R^2 \in [0,1]$ only holds for a least-squares fit *with an intercept*. **PICTURE.** Three mini-panels, same dots, three different lines: perfect (teal, $R^2=1$), flat-mean (grey, $R^2=0$), and a wild bad line (plum, $R^2<0$) whose red error sticks visibly dwarf the baseline sticks. --- ## Step 7 — Why $R^2$ alone lies: the adjusted fix **WHAT.** Add another feature — even a column of pure random noise — and refit. The line gains freedom to wiggle toward the dots, so $SS_{res}$ can only go **down or stay equal**, never up. Therefore $R^2$ can only **rise**. It rewards you for junk features. (This is the doorway to [[Overfitting]] and why we need [[Feature Selection]].) **WHY.** The cure is to stop comparing raw sums and instead compare *average* squared error, dividing each sum by its **degrees of freedom** — the number of independent pieces of information left after the model spent some fitting parameters. With $n$ points and $p$ predictors plus one intercept, the residuals have $n-p-1$ degrees of freedom; the total has $n-1$. $$R^2_{adj} \;=\; 1 - \frac{\,SS_{res}/(n-p-1)\,}{\,SS_{tot}/(n-1)\,} \;=\; 1 - (1-R^2)\,\underbrace{\frac{n-1}{n-p-1}}_{\text{penalty}>1\text{ and grows with }p}$$ Each new feature bumps $p$ up, shrinks $n-p-1$, and inflates the penalty factor. Unless the feature cuts $SS_{res}$ by *more* than the penalty grows, $R^2_{adj}$ **falls** — punishing you for noise while rewarding genuine signal. This is the metric [[Model Selection]] and [[Cross-Validation]] lean on when navigating the [[Bias-Variance Tradeoff]]. **PICTURE.** Two curves against the number of features $p$: plain $R^2$ climbing forever (orange), and $R^2_{adj}$ rising then turning back down (teal) once useless features enter. The gap between them *is* the penalty. --- ## The one-picture summary Everything above, compressed: the total spread bar splits into explained (teal) and residual (red); $R^2$ is the teal fraction; the adjusted version shaves the teal down by the degrees-of-freedom penalty when features pile up. > [!formula] The two boxes to carry away > $$R^2 = 1 - \frac{SS_{res}}{SS_{tot}}, \qquad R^2_{adj} = 1 - (1-R^2)\frac{n-1}{n-p-1}$$ > [!recall]- Feynman retelling — say it back in plain words > Imagine every data point drops a vertical stick to a flat line at the average — square those sticks, add them, and that whole pile is the *total spread* $SS_{tot}$, the size of the job. Now draw your model's line and measure the leftover sticks from each dot down to *your* line: square and add those to get $SS_{res}$, the mess you couldn't clean up. $R^2$ just asks "what fraction of the original pile did I clear?" — it's one minus (leftover mess ÷ total pile). Clear it all and you score $1$; clear nothing and you score $0$; make things worse than the average and you go negative. The catch: piling on extra features can only ever shrink the leftover pile, so raw $R^2$ can be fooled into looking better. Adjusted $R^2$ fixes this by comparing *average* error instead of total error, dividing by how many independent facts you have left after spending some on parameters — so junk features, which eat a degree of freedom without cleaning much mess, actually drag the score *down*. > [!recall]- Quick self-check > $SS_{tot}$ splits into which two piles? ::: The explained part $SS_{reg}$ and the unexplained residual part $SS_{res}$. > What does $R^2 = 0$ mean geometrically? ::: The model's line is no better than the flat baseline at $\bar{y}$; every prediction equals the mean. > Why can raw $R^2$ never decrease when you add a feature? ::: More freedom lets the line fit the dots at least as well, so $SS_{res}$ can only stay equal or shrink, pushing $R^2$ up. > What makes $R^2_{adj}$ drop for a useless feature? ::: The penalty $\frac{n-1}{n-p-1}$ grows with $p$, but a useless feature barely lowers $SS_{res}$, so the penalty wins.