Foundations — Linear regression — least squares, inference on coefficients

Before you can read the parent note, you need to own every squiggle it writes. Below, each symbol gets three things: plain words, a picture, and why the topic needs it. They are ordered so each one leans only on the ones above it.

1. The raw ingredients: dots on paper

Picture: each pair is a single dot pinned to graph paper. Read $x_i$ off the bottom, $y_i$ off the side.

Why the topic needs it: the whole enterprise is "find the line through a cloud of dots." No dots, nothing to fit.

Picture: imagine walking along the dots one by one, dropping each dot's value into a running total.

Why: every formula in the topic is a total taken over all dots — the sign just saves us writing "$+\cdots +$".

2. Where the cloud sits: the means

Picture: $\bar{x}$ is the horizontal balance-point of the dots (ignore heights); $\bar{y}$ is the vertical balance-point. The pair $(\bar{x},\bar{y})$ marks the centre of mass — the spot where the cloud would balance on a pin.

Why: the best-fit line is forced to pass through $(\bar{x},\bar{y})$. Knowing the centre first makes every later formula simpler, because we measure everything relative to this centre.

Picture: slide your graph so the centre of mass sits at the origin. Now each dot's coordinates are its centred values — a dot in the upper-right has both positive, a dot in the lower-left has both negative.

Why: the sign of these centred values is the engine of the slope (next section). Dots that are right-and-high or left-and-low will pull the line upward.

3. Spread and co-movement: $S_{xx}$ and $S_{xy}$

Picture: for every dot draw a horizontal stick from the centre line to the dot's $x$-position; $S_{xx}$ is the total of the squared stick lengths. Dots far out left/right make it large; dots bunched in the middle make it small.

Why: a wide horizontal spread is a long lever — it lets you aim the line's tilt precisely. This exact quantity later sits under the slope's precision.

Picture (crucial): the sign of each product tells a story.

Why: this tally is the numerator of the slope. It is a close cousin of Covariance (which is $S_{xy}/n$) and of the Correlation coefficient (which rescales it to lie in $[-1,1]$).

4. The line and its parts: ${\beta }_0,{\beta }_1$, hats, and residuals

Picture: stand at $x=0$; the line is at height ${\beta }_0$. Walk one step right; the line climbs by ${\beta }_1$.

Picture: the true line is invisible; the hatted line is the one we actually draw through the cloud.

Picture: $e_i$ is the length of the vertical segment from the dot down (or up) to the fitted line — the "miss." ${\epsilon }_i$ is the same idea but measured against the invisible true line.

5. The quantity we minimise, and the slope formula it produces

Picture: each residual stick from figure s04 gets turned into a square (side = the miss); $S$ is the total area of all those squares. A better-aimed line makes the squares smaller, so $S$ drops.

Why: least squares is defined as "pick ${\hat{\beta }}_0,{\hat{\beta }}_1$ that make $S$ as small as possible." So $S$ is the target — the very thing every derivation is pushing downhill. The prose "minimise the total squared residuals" and the symbol $S$ mean exactly the same thing.

Picture: $S$ is a bowl-shaped surface over the $({\beta }_0,{\beta }_1)$ plane; the minimum is the lowest point of the bowl, where the ground is flat in every direction.

Setting both partials to zero and solving pins down the estimates in terms of the pieces we already built:

Why this matters now: the parent topic derives these boxed formulas by calculus, but you should recognise every symbol in them before the derivation — and know the one input configuration ($S_{xx}=0$) that breaks them.

6. Trust language: variance, ${\sigma }^2$, standard error, and the $t$

Picture: a fat vertical band of scatter around the true line means large $\sigma$; a thin band means small $\sigma$.

Picture: overlay the many lines you'd get from many repeats; they form a fan. A narrow fan = small SE = trustworthy slope; a wide fan = large SE.

Why: dividing the summed squared residuals ($S$ again!) by $n-2$ (not $n$) gives an unbiased estimate of ${\sigma }^2$.

Why: because we had to estimate $\sigma$ from the same small sample, the bell curve gets fatter tails — that fattened curve is the $t$-distribution.

The claim that this fit is the best possible unbiased linear one comes from the Gauss–Markov Theorem; extending from one input $x$ to several is the job of Multiple Regression.

Prerequisite map

Everything on the left builds the line; everything on the right builds the trust in that line. Together they are exactly the two questions the parent topic answers.

Equipment checklist

Cover the right side and test yourself.