Foundations — Simple linear regression model

Before you can derive that line (which the parent note does), you must be fluent in the tiny alphabet of symbols it throws at you. This page defines every single one from zero — plain words first, then a picture, then why the topic can't live without it. Nothing here assumes you've seen algebra beyond "a line has a slope".

What a data point even is

The whole subject starts with paired measurements. You measure two numbers about the same thing and write them together.

Why the topic needs this. Everything else — the line, the errors, the formulas — is built on top of these dots. If you can plot a dot at horizontal position $x$ and height $y$, you already understand the raw material.

Why is $x$ horizontal and $y$ vertical? Because we chose $x$ to be the thing we know and $y$ the thing we compute from it. Horizontal = "what I feed in", vertical = "what comes out". This convention never changes in the whole chapter.

The straight line: slope and intercept

A straight line is described by exactly two numbers. Not three, not one — two. Give me those two numbers and I can draw the whole infinite line.

Slope is a ratio. Move right by $\Delta x$ (a change in $x$), the line rises by $\Delta y$. Then

${\beta }_1=\frac{\Delta y}{\Delta x}=\frac{\text{rise}}{\text{run}}.$

The triangle symbol $\Delta$ ("delta") just means "the change in". If price rises $20k every time size grows by 1 unit, the slope is $20$. This is exactly the "price per square foot" interpretation the parent note gives.

All cases of slope:

• ${\beta }_1>0$: line goes up to the right (bigger $x$ → bigger $y$).
• ${\beta }_1<0$: line goes down to the right (bigger $x$ → smaller $y$).
• ${\beta }_1=0$: flat line — $y$ ignores $x$ entirely. The best prediction is just a constant.

Prediction vs. reality: the hat and the residual

Here is the subtlety that trips everyone up. There are two $y$-values at every input.

They almost never match. The gap between them is the star of the whole show:

Why vertical and not perpendicular? Because we only ever make mistakes in $y$ — the thing we predict. The input $x$ is given to us for free; we never guess it. So "how wrong were we?" is measured straight up and down.

Why the topic needs residuals. The "best" line is defined as the one that makes these gaps small all together. No residuals, no notion of "best". This is the seed of Ordinary Least Squares in the parent note.

The error term $ϵ$ (Greek "epsilon") in the model $y={\beta }_0+{\beta }_{1}x+ϵ$ is the idealised cousin of the residual: it's the built-in randomness of the real world that no straight line could ever capture (measurement noise, a thousand tiny unmeasured causes). The residual $e_i$ is what we actually see; $ϵ$ is what we assume exists.

The summation sign: adding across all points

To score a whole line we must combine all the residuals into one number. That needs the sigma symbol.

Worked micro-example. If your residuals are $e_1=3$, $e_2=-1$, $e_3=2$ then ${\sum }_{i=1}^3{e}_i=3+(-1)+2=4.$

Why the topic needs it. The loss the parent minimises is $\text{RSS}=\sum (y_i-{\hat{y}}_i{)}^2$. Without $\sum$ we could only talk about one point at a time — but a good line is a compromise across all of them.

The mean: the balance point

Picture the dots' shadows on the horizontal axis; $\bar{x}$ is the point where they'd perfectly balance on a see-saw. Same for $\bar{y}$ vertically. Together $(\bar{x},\bar{y})$ is the centre of the whole cloud — and a beautiful fact the parent derives is that the best-fit line always passes through this centre point.

Spread and co-movement: variance and covariance

The slope formula in the parent is ${\beta }_1=\frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x}{)}^2}$. Two ideas hide inside it.

Squaring and the square root

Two humble operations do heavy lifting.

How it all fits together

Every node here is a symbol you just learned. Follow the arrows and you have reconstructed the entire logic of the parent topic — from a single dot to a fitted, scored model. This same foundation carries you forward into multiple regression, the optimisation view in gradient descent, the assumption checks, and eventually the curved cousin in logistic regression. Squaring residuals is also the seed of the overfitting story.

Equipment checklist

Cover the right side and answer each aloud before moving to the parent derivation.