Linear regression — least squares, inference on coefficients: Exercises

Before we start, we agree on the toolbox (nothing used before it is named):

A shared dataset (call it Set A) is used by several problems so you build fluency on one table:

$x = (1, 2, 3, 4, 5, 6), y = (3, 4, 6, 5, 8, 9) .$

The figure shows Set A with its least-squares line (cyan), its centroid $(\overset{x}{ˉ}, \overset{y}{ˉ})$ (amber cross), and the residuals $e_{i}$ as amber vertical segments. Keep it open while you work L2 and L3 — the line is what you fit, the segments are what you square to get SSE.

Figure — Linear regression — least squares, inference on coefficients

Level 1 — Recognition

L1.1 — Read the pieces off a line

The fitted line from some data is $\overset{y}{^} = 2.2 + 0.6 x$ . (a) What is the estimated slope $\hat{β}_{1}$ ? (b) What is the estimated intercept $\hat{β}_{0}$ ? (c) What does the slope mean in words?

Recall Solution

(a) The slope is the number multiplying $x$ : $\hat{β}_{1} = 0.6$ . (b) The intercept is the standalone number (value when $x = 0$ ): $\hat{β}_{0} = 2.2$ . (c) "Each time $x$ goes up by $1$ , the fitted $y$ goes up by $0.6$ on average." The slope is a rate, not a total.

L1.2 — Which number is which?

For Set A you are told $\overset{x}{ˉ} = 3.5$ . Compute $\overset{y}{ˉ}$ .

Recall Solution

$\overset{y}{ˉ} = \frac{3 + 4 + 6 + 5 + 8 + 9}{6} = \frac{35}{6} \approx 5.833$ . The mean is just add-them-up-and-divide; $n = 6$ here.

Level 2 — Application

L2.1 — Fit the line to Set A by hand

Using $x = (1, 2, 3, 4, 5, 6)$ , $y = (3, 4, 6, 5, 8, 9)$ , compute $\overset{x}{ˉ}, \overset{y}{ˉ}, S_{xx}, S_{x y}$ , then $\hat{β}_{1}$ and $\hat{β}_{0}$ .

Recall Solution

Means first (everything centres on them): $\overset{x}{ˉ} = 21/6 = 3.5$ , $\overset{y}{ˉ} = 35/6 \approx 5.8333$ .

Centred $x$ deviations $(x_{i} - \overset{x}{ˉ})$ : $- 2.5, - 1.5, - 0.5, 0.5, 1.5, 2.5$ . $S_{xx} = 2. 5^{2} + 1. 5^{2} + 0. 5^{2} + 0. 5^{2} + 1. 5^{2} + 2. 5^{2} = 6.25 + 2.25 + 0.25 + 0.25 + 2.25 + 6.25 = 17.5.$

Centred $y$ deviations $(y_{i} - \overset{y}{ˉ})$ : $- 2.8333, - 1.8333, 0.1667, - 0.8333, 2.1667, 3.1667$ . Multiply pairwise and add: $S_{x y} = (- 2.5) (- 2.8333) + (- 1.5) (- 1.8333) + (- 0.5) (0.1667)$ $+ (0.5) (- 0.8333) + (1.5) (2.1667) + (2.5) (3.1667) = 20.5.$ (Term by term: $7.0833 + 2.75 - 0.0833 - 0.4167 + 3.25 + 7.9167 = 20.5$ .)

Slope = co-movement over $x$ -spread: $\hat{β}_{1} = \frac{20.5}{17.5} \approx 1.1714$ . Intercept (forces the line through $(\overset{x}{ˉ}, \overset{y}{ˉ})$ ): $\hat{β}_{0} = 5.8333 - 1.1714 (3.5) \approx 1.7333$ . Fitted line: $\overset{y}{^} \approx 1.733 + 1.171 x$ .

L2.2 — Predict and check the centroid

Using the L2.1 line, predict $\overset{y}{^}$ at $x = 4$ , and verify the line passes through $(\overset{x}{ˉ}, \overset{y}{ˉ}) = (3.5, 5.8333)$ .

Recall Solution

At $x = 4$ : $\overset{y}{^} = 1.7333 + 1.1714 (4) = 1.7333 + 4.6857 = 6.4190$ . At $x = 3.5$ : $\overset{y}{^} = 1.7333 + 1.1714 (3.5) = 1.7333 + 4.1 = 5.8333 = \overset{y}{ˉ}$ ✓. The least-squares line always passes through the centre of mass — a free sanity check.

Level 3 — Analysis

L3.1 — Standard error and the $t$ -statistic

For Set A ( $n = 6$ ) the residuals are $e_{i} = y_{i} - \overset{y}{^}_{i}$ and the residual sum of squares is $SSE = \sum e_{i}^{2} = 7.31429$ (you may take this). Compute $s^{2}$ , $s$ , $SE (\hat{β}_{1})$ , and the $t$ -statistic for testing $β_{1} = 0$ . (These inference steps assume the error conditions stated in the toolbox above.)

Recall Solution

Residual variance (divide by $n - 2$ because two coefficients were estimated): $s^{2} = \frac{SSE}{n - 2} = \frac{7.31429}{4} = 1.82857, s = 1.82857 \approx 1.35225.$ Standard error of the slope (defined in the toolbox — precision improves with more $x$ -spread): $SE (\hat{β}_{1}) = \frac{s}{S _{xx}} = \frac{1.35225}{17.5} = \frac{1.35225}{4.18330} \approx 0.32324.$ $t$ -statistic (how many SEs the slope sits from zero): $t = \frac{β ^ _{1} - 0}{SE ( β ^ _{1} )} = \frac{1.17143}{0.32324} \approx 3.624 on n - 2 = 4 df .$

L3.2 — Decide significance

Critical value $t_{4, 0.975} = 2.776$ . At the 5% level, do we reject $β_{1} = 0$ ? Build the 95% CI.

Recall Solution

$∣ t ∣ = 3.624 > 2.776$ , so we reject $β_{1} = 0$ : the slope is significant. 95% CI: $\hat{β}_{1} \pm t_{4, 0.975} \cdot SE = 1.17143 \pm 2.776 (0.32324)$ $= 1.17143 \pm 0.89732 = [0.274, 2.069]$ . The interval excludes $0$ — consistent with rejection. Notice it's the same decision from two angles (Hypothesis Testing via $t$ , or via the CI using the t-distribution with $4$ df).

Level 4 — Synthesis

L4.1 — $R^{2}$ from the decomposition

For Set A, $S_{y y} = SST = 31.3333$ and $SSE = 7.31429$ . Compute $R^{2}$ two ways (via SSE and via SSR) and confirm $R^{2} = r^{2}$ , where $r$ is the Correlation coefficient.

Recall Solution

Total up–down variation: $SST = 31.3333$ . Residual (unexplained): $SSE = 7.31429$ . Explained: $SSR = SST - SSE = 31.3333 - 7.31429 = 24.0190$ .

R^2=\frac{\text{SSR}}{\text{SST}}=\frac{24.0190}{31.3333}=0.7666.\ \checkmark$$ **Correlation check (exact):** $r^2 = \dfrac{S_{xy}^2}{S_{xx}S_{yy}}=\dfrac{20.5^2}{17.5\cdot31.3333}=\dfrac{420.25}{548.33}=0.7664.$ The tiny gap from $0.7666$ is only rounding in the "given" SSE; with exact arithmetic $R^2=r^2$ identically. The line explains about **77\%** of the variance in $y$.

L4.2 — Forecast-then-verify the lever effect

Without recomputing from data: suppose you could triple the spread of the $x$ 's (so $S_{xx}$ becomes $9 \times$ larger) while $s$ stays the same. Predict what happens to $SE (\hat{β}_{1})$ , then confirm with the formula. What does this say physically?

Recall Solution

Predict: $SE (\hat{β}_{1}) = s / S_{xx}$ . If $S_{xx} \to 9 S_{xx}$ , then $S_{xx} \to 3 S_{xx}$ , so the SE should shrink to one-third. Verify numerically: old SE $= 1.3523/ 17.5 = 0.3233$ ; new SE $= 1.3523/ 157.5 = 0.1078$ , and $0.3233/0.1078 = 3.00$ ✓. Physical meaning: wide-spread $x$ -values give the line a long lever arm, so the same vertical wobble pins the slope down far more precisely — echoing the Gauss–Markov Theorem idea that spread-out design points make the least-squares estimator sharp.

Level 5 — Mastery

L5.1 — Rebuild the slope's variance from scratch

Starting from $\hat{β}_{1} = \sum_{i} c_{i} y_{i}$ with weights $c_{i} = (x_{i} - \overset{x}{ˉ}) / S_{xx}$ , and using $Var (y_{i}) = σ^{2}$ with independent errors, prove that $Var (\hat{β}_{1}) = σ^{2} / S_{xx}$ . Show why the $c_{i}$ collapse so neatly.

Recall Solution

What we do: treat $\hat{β}_{1}$ as a fixed weighted sum of random $y_{i}$ . For independent terms, variance of a sum is the sum of (weight $^{2} \times$ variance): $Var (\hat{β}_{1}) = \sum_{i} c_{i}^{2} Var (y_{i}) = σ^{2} \sum_{i} c_{i}^{2} .$ Why: independence kills all the cross-covariance terms — only the diagonal survives. Now the collapse:

=\frac{S_{xx}}{S_{xx}^2}=\frac{1}{S_{xx}}.$$ Therefore $\text{Var}(\hat\beta_1)=\dfrac{\sigma^2}{S_{xx}}$. The numerator sum-of-squares *is* $S_{xx}$, so one power cancels. This is exactly the "long lever" fact made rigorous, and replacing $\sigma^2$ by $s^2$ gives the $\text{SE}(\hat\beta_1)=s/\sqrt{S_{xx}}$ we assumed in L3. (Under Gaussian errors this variance, plugged into a $t$-ratio, is what [[Maximum Likelihood Estimation]] would also hand you.)

L5.2 — Full pipeline on new data

New dataset Set B: $x = (0, 1, 2, 3, 4)$ , $y = (1, 1, 2, 2, 4)$ . Do the whole job: $\hat{β}_{1}, \hat{β}_{0}$ , $SSE$ , $s^{2}$ , $SE (\hat{β}_{1})$ , the $t$ -statistic for $β_{1} = 0$ , and the significance decision at 5% ( $t_{3, 0.975} = 3.182$ ).

Recall Solution

Means: $\overset{x}{ˉ} = 2$ , $\overset{y}{ˉ} = (1 + 1 + 2 + 2 + 4) /5 = 2$ . $x$ -deviations: $- 2, - 1, 0, 1, 2$ → $S_{xx} = 4 + 1 + 0 + 1 + 4 = 10$ . $y$ -deviations: $- 1, - 1, 0, 0, 2$ . $S_{x y} = (- 2) (- 1) + (- 1) (- 1) + 0 + 0 + (2) (2) = 2 + 1 + 0 + 0 + 4 = 7$ . Slope: $\hat{β}_{1} = 7/10 = 0.7$ . Intercept: $\hat{β}_{0} = 2 - 0.7 (2) = 0.6$ . Line $\overset{y}{^} = 0.6 + 0.7 x$ .

Fitted values: $0.6, 1.3, 2.0, 2.7, 3.4$ . Residuals $e_{i} = y_{i} - \overset{y}{^}_{i}$ : $0.4, - 0.3, 0.0, - 0.7, 0.6$ . $SSE = 0.16 + 0.09 + 0 + 0.49 + 0.36 = 1.10.$ $s^{2} = \frac{1.10}{5 - 2} = 0.3667$ , $s = 0.6055$ . $SE (\hat{β}_{1}) = \frac{0.6055}{10} = 0.1915$ . $t = \frac{0.7}{0.1915} = 3.656$ on $3$ df. Since $3.656 > 3.182$ , we reject $β_{1} = 0$ at 5%: the slope is significant even with only 5 points, because the fit is tight and the $x$ 's are spread.

Recall One-line self-test

Slope of least squares ::: $\hat{β}_{1} = S_{x y} / S_{xx}$ Why divide SSE by $n - 2$ ::: two constraints ( $\sum e_{i} = 0$ , $\sum x_{i} e_{i} = 0$ ) remove 2 df, making $s^{2}$ unbiased What makes $SE (\hat{β}_{1})$ small ::: large $S_{xx}$ (spread-out $x$ ) and small $s$ (little noise) $R^{2}$ in one phrase ::: fraction of $y$ 's variance explained by the line, equals $r^{2}$ here