Visual walkthrough — Linear regression — least squares, inference on coefficients

Step 1 — The dots and the guess

WHAT. We start with a handful of measured pairs: for each observation $i$ we have a horizontal number $x_i$ and a vertical number $y_i$. Plotting them gives a cloud of points. We suspect the cloud secretly follows a straight line, but noise has jostled every dot.

WHY. A straight line is the simplest possible "story": pick a starting height and a steepness, and you've described infinitely many points with just two numbers. Before we can find the best line we must be able to draw any line and score it.

PICTURE. Below, the black dots are the data. The blue line is one candidate — a guess. Its two knobs are:

$\hat{y}=\underset{\text{height where line crosses }x=0}{\underbrace{{\hat{\beta }}_0}}+\underset{\text{steepness: rise per 1 step right}}{\underbrace{{\hat{\beta }}_1}}\cdot x$

Step 2 — Measuring a miss: the residual

WHAT. For each dot, drop a vertical segment from the dot to the line. Its signed length is the residual:

$e_i=\underset{\text{where the dot IS}}{\underbrace{y_i}}-\underset{\text{where the line SAYS it should be}}{\underbrace{{\hat{y}}_i}}$

WHY vertical, not perpendicular? We are trying to predict $y$ from $x$. Given an $x$, the line hands us a predicted $y$. The honest error is the gap in the thing we predict — the $y$-direction. Perpendicular distance would mix errors in $x$ (which we treat as known) into the score.

PICTURE. Orange segments are the misses. Dots above the line have $e_i>0$; dots below have $e_i<0$.

Step 3 — Why we square the misses

WHAT. We score a line by the sum of squared residuals:

$S({\hat{\beta }}_0,{\hat{\beta }}_1)={\sum }_{i=1}^n{e}_i^2={\sum }_{i=1}^n(\underset{\text{the miss for dot }i}{\underbrace{y_i-{\hat{\beta }}_0-{\hat{\beta }}_1{x}_i}}{)}^2$

WHY squaring, and not absolute value? Three reasons, each visible:

1. Squaring turns every miss positive, so nothing cancels.
2. Squaring punishes big misses harder — a miss of $2$ costs $4$, a miss of $4$ costs $16$. The line is pulled hardest by its worst offences.
3. Squaring is smooth: the score curve has no sharp corner, so calculus (Step 5) can find the bottom cleanly. Absolute value has a kink and gives no tidy formula.

PICTURE. Each orange segment of length $e_i$ becomes an orange square of area $e_i^2$. The total shaded area is the score $S$. Least squares = shrink the total orange area.

Step 4 — The score lives in a bowl

WHAT. $S$ depends on the two knobs ${\hat{\beta }}_0,{\hat{\beta }}_1$. Treat those two numbers as a location and plot $S$ as a height above them. Because $S$ is a sum of squares, that surface is a smooth upward bowl (a paraboloid) with exactly one lowest point.

WHY this matters. A bowl has a single bottom, and at that bottom the ground is flat in every direction. "Flat" is something calculus can pin down exactly. This is what guarantees a unique best line — no second candidate, no ambiguity.

PICTURE. The bowl over the $({\hat{\beta }}_0,{\hat{\beta }}_1)$ plane. The red dot is the minimum; the red tangent plane there is horizontal.

Step 5 — Set the two slopes of the bowl to zero

WHAT. We take the derivative of $S$ with respect to each knob and set both to zero. The derivative $\partial S/\partial {\hat{\beta }}_0$ asks "if I nudge the height, does the score change?"; at the bottom the answer must be "no."