Worked examples — Linear regression — least squares, inference on coefficients

Before starting, recall the five numbers everything is built from — computed once, reused everywhere. Throughout, $n$ is the number of data points (how many $(x_i,y_i)$ pairs we have); the little subscript $i$ just labels which point we mean, running $i=1,2,\dots ,n$.

Recall The five quantities you always compute first

$\bar{x}$ (mean of $x$) ::: the "centre" of your inputs along the horizontal $\bar{y}$ (mean of $y$) ::: the "centre" of your outputs along the vertical $S_{xx}=\sum (x_i-\bar{x}{)}^2$ ::: how spread out the $x$'s are (always $\ge 0$) $S_{xy}=\sum (x_i-\bar{x})(y_i-\bar{y})$ ::: how $x$ and $y$ co-move (any sign) $S_{yy}=\sum (y_i-\bar{y}{)}^2$ ::: total spread of the $y$'s (this is SST)

Here $\sum$ (the capital Greek "sigma") just means "add up over all $n$ data points," and a bar over a letter like $\bar{x}$ means "the average of that letter."

---

The scenario matrix

Every simple-regression problem lives in exactly one of these cells. The last column names the example that nails it.

#	Case class	What's special	Watch out for	Example
A	Positive slope	$S_{xy}>0$, points trend up	ordinary case	Ex 1
B	Negative slope	$S_{xy}<0$, points trend down	sign of ${\hat{\beta }}_1$	Ex 2
C	Zero slope	$S_{xy}=0$ exactly	line is flat, ${\hat{\beta }}_1=0$, $R^2=0$	Ex 3
D	Degenerate $x$	all $x_i$ equal, $S_{xx}=0$	division by zero — no line	Ex 4
E	Perfect fit	points exactly on a line, SSE $=0$	$R^2=1$, SE $=0$, $t=\infty$	Ex 5
F	Constant response	all $y_i$ equal, $S_{yy}=0$	$R^2=0/0$ undefined	Ex 6
G	Real word problem	units, prediction, interpretation	keep units on every number	Ex 7
H	Inference / significance	is the slope real?	$t$ vs critical value, CI	Ex 8
I	Exam twist: rescale $x$	change units of $x$	how ${\hat{\beta }}_1$, SE, $R^2$ move	Ex 9

We now walk them in order. Cells A–C share one picture so you can see how the sign of $S_{xy}$ flips the tilt.

Figure s01 — the same five $x$-values with three different $y$-patterns. Left (magenta, Cell A): points climb, the fitted navy line tilts up. Middle (violet, Cell B): points fall, the line tilts down. Right (orange, Cell C): points balance out, the line is flat. In all three the star marks $(\bar{x},\bar{y})$ — notice every line passes exactly through it.

---

Example 1 — Cell A: positive slope

Step 1 — centre the data. $\bar{x}=\frac{1+2+3+4+5}{5}=3$, $\bar{y}=\frac{1+3+2+5+4}{5}=3$. Why this step? The whole formula measures how points deviate from their centre; the line is forced through $(\bar{x},\bar{y})$, so we anchor there first.

Step 2 — spread of $x$. $S_{xx}=(-2{)}^2+(-1{)}^2+0^2+1^2+2^2=4+1+0+1+4=10.$ Why? This is the "lever length" that will pin the slope down later.

Step 3 — co-movement. Deviations of $y$: $(-2,0,-1,2,1)$. $S_{xy}=(-2)(-2)+(-1)(0)+(0)(-1)+(1)(2)+(2)(1)=4+0+0+2+2=8.$ Why? $S_{xy}$ pairs each $x$-deviation with its $y$-deviation. Both positive or both negative → positive product → upward trend.

Step 4 — slope and intercept.

\hat\beta_0=\bar y-\hat\beta_1\bar x=3-0.8(3)=0.6.$$ Line: $\hat y=0.6+0.8x$. > **Verify:** Passes through $(\bar x,\bar y)=(3,3)$? $0.6+0.8(3)=0.6+2.4=3$ ✓. Slope $0.8>0$ and > below 1, matching the forecast. --- ## Example 2 — Cell B: negative slope > [!example] Statement > Same $x=(1,2,3,4,5)$ but now $y=(5,4,4,2,1)$. Fit the line. > > **Forecast:** Dots fall left-to-right, so $\hat\beta_1<0$. The intercept (value at $x=0$, off to > the left) should be *above* $\bar y$. **Step 1 — centres.** $\bar x=3$, $\bar y=\dfrac{5+4+4+2+1}{5}=\dfrac{16}{5}=3.2$. **Step 2 — $S_{xx}=10$** (same $x$ as before — reuse it). *Why reuse?* $S_{xx}$ depends on $x$ only. **Step 3 — co-movement.** $y$-deviations: $(1.8,0.8,0.8,-1.2,-2.2)$. $$S_{xy}=(-2)(1.8)+(-1)(0.8)+0(0.8)+1(-1.2)+2(-2.2)=-3.6-0.8+0-1.2-4.4=-10.$$ *Why negative?* Left points sit above centre, right points below — opposite signs multiply to negative. That negative $S_{xy}$ *is* the downward tilt. **Step 4 — slope and intercept.** $$\hat\beta_1=\frac{-10}{10}=-1,\qquad \hat\beta_0=3.2-(-1)(3)=6.2.$$ Line: $\hat y=6.2-x$. > **Verify:** Through $(3,3.2)$? $6.2-3=3.2$ ✓. Slope $-1<0$; intercept $6.2>\bar y=3.2$ — both match > the forecast. --- ## Example 3 — Cell C: exactly zero slope > [!example] Statement > $x=(1,2,3,4,5)$, $y=(2,4,3,4,2)$ — a symmetric hump. Fit the line and find $R^2$. > > **Forecast:** The pattern rises then falls with no overall tilt; the "best line" should be **flat** > at the mean of $y$, giving $\hat\beta_1=0$ and $R^2=0$ (a flat line explains no linear variation). **Step 1 — centres.** $\bar x=3$, $\bar y=\dfrac{2+4+3+4+2}{5}=3$. **Step 2 — $S_{xx}=10$**, and $y$-deviations $(-1,1,0,1,-1)$. **Step 3 — co-movement.** $$S_{xy}=(-2)(-1)+(-1)(1)+(0)(0)+(1)(1)+(2)(-1)=2-1+0+1-2=0.$$ *Why zero?* The left side ($x<\bar x$) and right side ($x>\bar x$) contribute equal and opposite amounts — they cancel exactly. No linear co-movement at all. **Step 4 — estimates.** $\hat\beta_1=0/10=0$, $\hat\beta_0=3-0=3$. Line: $\hat y=3$ (flat). **Step 5 — goodness of fit.** With $\hat y_i=\bar y$ for all $i$, every fitted value equals the mean, so the explained sum $\text{SSR}=\sum(\hat y_i-\bar y)^2=0$, giving $$R^2=\frac{\text{SSR}}{\text{SST}}=0.$$ > **Verify:** $\text{SST}=(-1)^2+1^2+0^2+1^2+(-1)^2=4$. $\text{SSE}=\sum(y_i-3)^2=4$ (line predicts $3$ > everywhere). Then $R^2=1-\text{SSE}/\text{SST}=1-4/4=0$ ✓. A flat line explains **0\%** of the > variation — the boundary between Cells A and B. --- ## Example 4 — Cell D: degenerate input, no line exists > [!example] Statement > $x=(2,2,2,2)$, $y=(1,3,2,4)$. Try to fit the line. > > **Forecast:** All the $x$'s are stacked on one vertical line. There's no "left-to-right" tilt to > measure — you can draw infinitely many lines through that stack. So the slope should be > **undefined**. **Step 1 — spread of $x$.** $\bar x=2$, so every deviation $x_i-\bar x=0$ and $$S_{xx}=0^2+0^2+0^2+0^2=0.$$ **Step 2 — attempt the slope.** $$\hat\beta_1=\frac{S_{xy}}{S_{xx}}=\frac{S_{xy}}{0}\quad\Rightarrow\quad \textbf{undefined}.$$ *Why this step matters:* division by $0$ isn't a huge number — it's *no answer*. Geometrically the "lever" has zero length, so the slope can't be aimed at all. > [!mistake] "$S_{xx}=0$ so just set the slope to 0." > *Why it feels right:* dividing by something tiny usually gives a definite value. **Fix:** here the > numerator's meaning ($x$-weighted co-movement) also collapses — there is genuinely *no* least-squares > line. You can only estimate a constant $\hat\beta_0=\bar y=2.5$. Statistical software typically > flags this — depending on the package you may get a warning, an `NA`/`NaN` slope, or an outright > error — but *never* a trustworthy finite slope. > **Verify:** $S_{xx}=0$, so $\hat\beta_1=S_{xy}/S_{xx}$ has a zero denominator — undefined ✓. See the > vertical stack in the figure below. ![[deepdives/dd-maths-4.9.22-d3-s02.png]] *Figure s02 — Cell D. All four points share $x=2$, so they stack on one vertical dashed line. The star marks $(\bar x,\bar y)=(2,2.5)$. Three coloured dashed lines through the star all "fit" equally well — with no horizontal spread ($S_{xx}=0$) there is no way to prefer one tilt over another, so the slope is undefined.* --- ## Example 5 — Cell E: perfect fit, the limiting case > [!example] Statement > $x=(1,2,3,4)$, $y=(3,5,7,9)$ — every point sits exactly on a line. Fit it and find $R^2$, SSE, and > the standard error of the slope. > > **Forecast:** If the points are *exactly* linear there's zero leftover noise. Expect SSE $=0$, > $R^2=1$, and $\text{SE}(\hat\beta_1)=0$ (nothing to be uncertain about). **Step 1 — centres.** $\bar x=2.5$, $\bar y=\dfrac{3+5+7+9}{4}=6$. **Step 2 — sums.** $x$-deviations $(-1.5,-0.5,0.5,1.5)$, $y$-deviations $(-3,-1,1,3)$. $$S_{xx}=2.25+0.25+0.25+2.25=5,\qquad S_{xy}=4.5+0.5+0.5+4.5=10.$$ **Step 3 — line.** $\hat\beta_1=10/5=2$, $\hat\beta_0=6-2(2.5)=1$. Line: $\hat y=1+2x$. **Step 4 — residuals.** Predictions: $3,5,7,9$ — identical to the data. So every $e_i=0$, giving $\text{SSE}=0$. **Step 5 — goodness and precision.** $$R^2=1-\frac{\text{SSE}}{\text{SST}}=1-\frac{0}{20}=1,\qquad s^2=\frac{\text{SSE}}{n-2}=\frac{0}{2}=0,\qquad \text{SE}(\hat\beta_1)=\frac{s}{\sqrt{S_{xx}}}=0.$$ *Why the SE is zero:* with no scatter, every resampled dataset gives the *same* line — zero wobble. The $t$-statistic $\hat\beta_1/\text{SE}$ would be $2/0=\infty$: infinitely significant, as it should be. > **Verify:** SST $=9+1+1+9=20$; SSE $=0$; $R^2=1$ ✓; $s^2=0$; SE $=0$ ✓. This is the opposite extreme > from Cell C ($R^2=0$). --- ## Example 6 — Cell F: constant response (all $y$ equal) > [!example] Statement > $x=(1,2,3,4,5)$, $y=(7,7,7,7,7)$ — the output never changes. Fit the line and try to find $R^2$. > > **Forecast:** With $y$ flat, there is *nothing* for a slope to explain and *nothing* to be > explained. Expect $\hat\beta_1=0$, but $R^2$ to be **undefined** — you can't take a fraction of zero > total variation. **Step 1 — centres.** $\bar x=3$, $\bar y=7$. **Step 2 — sums.** $y$-deviations are all $0$, so $$S_{xy}=\sum(x_i-\bar x)(y_i-\bar y)=0,\qquad S_{yy}=\text{SST}=\sum(y_i-7)^2=0.$$ $S_{xx}=10$ (the $x$'s still spread). **Step 3 — slope and line.** $\hat\beta_1=0/10=0$, $\hat\beta_0=7-0=7$. The fitted line is $\hat y=7$, which *exactly* reproduces every data point, so $\text{SSE}=0$ too. **Step 4 — attempt $R^2$.** $$R^2=\frac{\text{SSR}}{\text{SST}}=\frac{0}{0}\quad\Rightarrow\quad \textbf{undefined}.$$ *Why this step matters:* $R^2$ answers "what fraction of $y$'s variation did the line capture?" — but here $y$ has **no** variation, so the question itself is empty ($0/0$). > [!mistake] "$R^2=0$ when $y$ is constant, since the line explains nothing." > *Why it feels right:* SSR $=0$, and $0/\text{something}=0$. **Fix:** the denominator SST is *also* > $0$, so $R^2=0/0$ is genuinely undefined, not $0$. Contrast Cell C, where SST $>0$ gave a clean > $R^2=0$. Inference is meaningless here too: $s^2=0$ so every SE is $0$ and the $t$-ratio is $0/0$. > **Verify:** $S_{yy}=0$ and $\text{SSR}=0$, so $R^2=\text{SSR}/\text{SST}=0/0$ — undefined ✓. The > slope $\hat\beta_1=0$ is well-defined because $S_{xx}=10\neq0$. --- ## Example 7 — Cell G: real-world word problem with units > [!example] Statement > A café records daily high temperature $x$ (in °C) and cups of iced coffee sold $y$: > $x=(20,22,24,26,28)$, $y=(30,35,40,48,52)$ cups. Fit the line and **predict sales at 25 °C**. > > **Forecast:** Hotter → more iced coffee, so $\hat\beta_1>0$. Roughly sales rise ~22 cups over an > 8 °C range, so the slope is around $22/8\approx 2.7$ cups per °C. **Step 1 — centres.** $\bar x=24$ °C, $\bar y=\dfrac{30+35+40+48+52}{5}=41$ cups. **Step 2 — sums.** $x$-deviations $(-4,-2,0,2,4)$, $y$-deviations $(-11,-6,-1,7,11)$. $$S_{xx}=16+4+0+4+16=40\ (\text{°C}^2),$$ $$S_{xy}=(-4)(-11)+(-2)(-6)+0(-1)+2(7)+4(11)=44+12+0+14+44=114\ (\text{°C·cups}).$$ **Step 3 — slope with units.** $$\hat\beta_1=\frac{114\ \text{°C·cups}}{40\ \text{°C}^2}=2.85\ \frac{\text{cups}}{\text{°C}}.$$ *Why track units?* The slope means "extra cups per extra degree." Getting the units right is the sanity check that you divided the correct pair. **Step 4 — intercept and prediction.** $$\hat\beta_0=41-2.85(24)=41-68.4=-27.4\ \text{cups}.$$ At $x=25$ °C: $\hat y=-27.4+2.85(25)=-27.4+71.25=43.85\approx 44$ cups. > [!mistake] "Intercept $=-27.4$ means the café sells negative coffee at 0 °C." > *Why it feels right:* $\hat\beta_0$ is the value at $x=0$. **Fix:** $x=0$ °C is far outside the data > range (20–28 °C). Extrapolating a line there is meaningless — the intercept is just a fitting > anchor, not a real forecast at freezing. > **Verify:** Slope $2.85$ cups/°C matches the ~2.7 forecast. Prediction at $\bar x=24$ should equal > $\bar y$: $-27.4+2.85(24)=41$ ✓. At $25$ °C, $43.85$ cups ✓ — just above the mean, as expected one > degree above centre. --- ## Example 8 — Cell H: is the slope real? (inference) > [!example] Statement > Reuse the café data (Ex 7). Test $H_0:\beta_1=0$ (temperature has no effect) at the 5\% level, and > give a 95\% confidence interval for the slope. > > **Forecast:** The upward trend looked very clean, so expect a large $t$-statistic and a CI well > above 0 — we should **reject** "no effect." **Step 1 — residuals.** With $\hat y=-27.4+2.85x$, at $x=20,22,24,26,28$ the fitted values are $29.6,\ 35.3,\ 41.0,\ 46.7,\ 52.4$. Subtracting from the data $y=(30,35,40,48,52)$ gives residuals $$e=y-\hat y=(0.4,\ -0.3,\ -1.0,\ 1.3,\ -0.4).$$ *Why residuals?* They are our only window onto the unseen noise, and that noise drives all the uncertainty in the slope. **Step 2 — residual variance $s^2$ (our estimate of $\sigma^2$).** $$\text{SSE}=0.4^2+(-0.3)^2+(-1.0)^2+1.3^2+(-0.4)^2=0.16+0.09+1.0+1.69+0.16=3.1.$$ $$s^2=\frac{\text{SSE}}{n-2}=\frac{3.1}{5-2}=\frac{3.1}{3}=1.0333,\qquad s=\sqrt{1.0333}=1.0165.$$ *Why divide by $n-2$ and not $n$?* We have $n=5$ points but spent 2 of them pinning down $\hat\beta_0$ and $\hat\beta_1$; only $n-2=3$ independent pieces of noise information remain. **Step 3 — standard error of the slope.** $$\text{SE}(\hat\beta_1)=\frac{s}{\sqrt{S_{xx}}}=\frac{1.0165}{\sqrt{40}}=\frac{1.0165}{6.3246}=0.16072.$$ *Why this formula?* The slope is a weighted sum of the noisy $y_i$; propagating their variance gives $\text{Var}(\hat\beta_1)=\sigma^2/S_{xx}$, and we plug in $s$ for the unknown $\sigma$. **Step 4 — $t$-statistic.** $$t=\frac{\hat\beta_1-0}{\text{SE}(\hat\beta_1)}=\frac{2.85}{0.16072}=17.73\quad\text{on }n-2=3\text{ df}.$$ *Why compare to a $t$-distribution?* Because we estimated $\sigma$ from the same small sample, the extra uncertainty fattens the tails — that is exactly what the [[t-distribution]] with $3$ degrees of freedom encodes (see [[Hypothesis Testing]]). **Step 5 — decision and 95\% confidence interval.** The critical value is $t_{3,\,0.975}=3.182$. Since $17.73>3.182$, we **reject** $H_0$: the effect is real. The interval is $$\hat\beta_1\pm t_{3,0.975}\cdot\text{SE}(\hat\beta_1)=2.85\pm 3.182(0.16072)=2.85\pm 0.5114=[2.34,\ 3.36]\ \text{cups/°C}.$$ > **Verify:** $\text{SSE}=3.1$; $s^2=3.1/3\approx1.0333$; $\text{SE}\approx0.1607$; $t\approx17.7$ ✓. > The CI $[2.34,3.36]$ excludes $0$, consistent with rejection — exactly matching the forecast that > this clean trend is significant. --- ## Example 9 — Cell I: exam twist — rescale the $x$-axis > [!example] Statement > In the café problem (Ex 7–8), an intern re-records temperature in **°F** instead of °C, so every > $x_i$ becomes $x_i'=1.8x_i+32$. **Without refitting from scratch**, how do $\hat\beta_1$, $R^2$, and > $\text{SE}(\hat\beta_1)$ change? > > **Forecast:** Each degree °F is a *smaller* step than a degree °C, so "cups per degree" should > **shrink** by the factor $1.8$. The *pattern* of the points is unchanged, so $R^2$ should stay the > same. The SE should shrink in the same proportion as the slope. **Step 1 — how the sums transform.** A linear rescale $x'=a x+b$ (here $a=1.8$, $b=32$) shifts and stretches deviations: $x_i'-\bar{x'}=a(x_i-\bar x)$. Therefore $$S_{x'x'}=a^2 S_{xx},\qquad S_{x'y}=a\,S_{xy}.$$ *Why this step?* The additive shift $b$ cancels in every deviation; only the multiplier $a$ survives, and it survives *twice* in $S_{x'x'}$ (two $x$-factors) but *once* in $S_{x'y}$. **Step 2 — new slope.** $$\hat\beta_1'=\frac{S_{x'y}}{S_{x'x'}}=\frac{a\,S_{xy}}{a^2 S_{xx}}=\frac{1}{a}\hat\beta_1 =\frac{2.85}{1.8}=1.5833\ \frac{\text{cups}}{\text{°F}}.$$ **Step 3 — $R^2$ is invariant.** $R^2=r^2$ where the [[Correlation coefficient|correlation]] $r=\dfrac{S_{xy}}{\sqrt{S_{xx}S_{yy}}}$. Under the rescale $r'=\dfrac{aS_{xy}}{\sqrt{a^2S_{xx}\,S_{yy}}}=\dfrac{a}{|a|}\,r=r$ (since $a=1.8>0$). So $R^2$ is **unchanged**. *Why?* Correlation ignores units — it's about pattern, not scale. **Step 4 — new standard error.** SSE and $s$ depend only on $y$ and the residuals, which don't change (the fitted values are identical, just relabelled on a stretched axis). So $s$ is unchanged, and $$\text{SE}(\hat\beta_1')=\frac{s}{\sqrt{S_{x'x'}}}=\frac{s}{a\sqrt{S_{xx}}}=\frac{1}{a}\text{SE}(\hat\beta_1) =\frac{0.16072}{1.8}=0.08929.$$ **Step 5 — the $t$-statistic is invariant.** $$t'=\frac{\hat\beta_1'}{\text{SE}(\hat\beta_1')}=\frac{\hat\beta_1/a}{\text{SE}(\hat\beta_1)/a} =\frac{\hat\beta_1}{\text{SE}(\hat\beta_1)}=t=17.73.$$ *Why this is beautiful:* changing units cannot change whether an effect is significant. The slope number moved, but the *evidence* did not. > **Verify:** $\hat\beta_1'=2.85/1.8\approx1.583$ ✓; $\text{SE}=0.1607/1.8\approx0.0893$ ✓; $t$ > unchanged at $\approx17.7$ ✓; $R^2$ identical. Slope and SE both scaled by $1/1.8$, exactly matching > the forecast. --- ## Where each cell used its tools ```mermaid flowchart TD A["Compute Sxx and Sxy"] --> B{"Is Sxx = 0?"} B -->|yes| C["Degenerate: no line, Cell D"] B -->|no| D["Slope = Sxy over Sxx"] D --> E{"Sign of Sxy"} E -->|positive| F["Up trend, Cell A"] E -->|zero| G["Flat line, R2 = 0, Cell C"] E -->|negative| H["Down trend, Cell B"] D --> I{"Is SST = 0?"} I -->|yes| J["Constant y, R2 undefined, Cell F"] I -->|no| K{"Is SSE = 0?"} K -->|yes| L["Perfect fit, R2 = 1, Cell E"] K -->|no| M["Inference: t and CI, Cell H"] ``` --- > [!recall]- Rapid self-test > If all $x_i$ are equal, what is $\hat\beta_1$? ::: Undefined — $S_{xx}=0$, division by zero (Cell D). > If all $y_i$ are equal, what is $R^2$? ::: Undefined — $\text{SST}=0$ gives $0/0$ (Cell F). > $S_{xy}=0$ exactly. What are $\hat\beta_1$ and $R^2$? ::: Both $0$; the best line is flat at $\bar y$ (Cell C). > Points lie exactly on a line. What is SE of the slope? ::: $0$ — SSE $=0$ so $s=0$ (Cell E). > You switch $x$ from cm to mm ($x'=10x$). What happens to the $t$-statistic? ::: Nothing — $t$ is unit-invariant (Cell I). > Positive $S_{xy}$ tells you the slope is…? ::: Positive — $x$ and $y$ co-move upward (Cell A). > [!mnemonic] The whole matrix in one breath > **"Sign of $S_{xy}$ sets the tilt; $S_{xx}=0$ kills the fit; $S_{yy}=0$ kills $R^2$; SSE $=0$ > perfects it; units move the number, never the $t$."** See also: [[Correlation coefficient]] (drives $R^2$), [[Covariance]] (the un-normalised $S_{xy}$), [[t-distribution]] and [[Hypothesis Testing]] (Cell H), [[Gauss–Markov Theorem]] (why least squares is best), [[Maximum Likelihood Estimation]] (why squared errors), and [[Multiple Regression]] (the same story with more $x$'s).