Visual walkthrough — Simple linear regression model

This page rebuilds the Simple Linear Regression result from nothing. No formula is borrowed. By the end you will have drawn your way to

${\beta }_1=\frac{\sum (x_i-\bar{x})(y_i-\bar{y})}{\sum (x_i-\bar{x}{)}^2},{\beta }_0=\bar{y}-{\beta }_1\bar{x}$

and you will know what every single symbol is doing on the picture.

---

Step 1 — The dots, and what a "line" even means here

WHAT. We start with $n$ measured points. Point number $i$ is written $(x_i,y_i)$: its horizontal position $x_i$ (the thing we know, e.g. size) and its vertical position $y_i$ (the thing we want, e.g. price). The little subscript $i$ is just a name tag — $i=1,2,\dots ,n$.

WHY. Before choosing a best line we must agree on what family of lines we are even choosing from. We restrict ourselves to straight lines because we believe "one more unit of $x$ adds a roughly fixed amount of $y$."

PICTURE. Look at the figure. A candidate straight line is

$\hat{y}=\underset{\text{where the line crosses }x=0}{\underbrace{{\beta }_0}}+\underset{\text{tilt: rise per 1 step right}}{\underbrace{{\beta }_1}}\cdot x$

The hat on $\hat{y}$ means predicted — the height the line claims at position $x$, as opposed to the real measured height $y_i$.

• ${\beta }_0$ (intercept) slides the whole line up and down.
• ${\beta }_1$ (slope) rotates it — steeper means each step right climbs more.

Two knobs ${\beta }_0,{\beta }_1$. Turning them gives every possible straight line. Our job: find the one good setting.

---

Step 2 — The residual: how wrong one point is

WHAT. For a chosen line, at each data point measure the vertical gap between the real dot and the line directly below/above it:

$e_i=\underset{\text{real height}}{\underbrace{y_i}}-\underset{\text{line’s guess}}{\underbrace{{\hat{y}}_i}}=y_i-({\beta }_0+{\beta }_1{x}_i)$

This gap $e_i$ is the residual.

WHY. We need a number that says "how bad is this line, here?" The natural badness at one point is how far the line missed the dot. We measure it vertically, not perpendicular, because $y$ is what we predict — a wrong price is a vertical error.

PICTURE. The red segment is $e_i$. If the dot sits above the line, $e_i>0$; below, $e_i<0$; exactly on it, $e_i=0$.

---

Step 3 — Squaring: turning gaps into a single bowl of cost

WHAT. Square each residual and sum them. This is the RSS (Residual Sum of Squares):

$\text{RSS}({\beta }_0,{\beta }_1)={\sum }_{i=1}^n{e}_i^2={\sum }_{i=1}^n(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2$

Term by term: $y_i-{\beta }_0-{\beta }_1{x}_i$ is the gap; $({)}^2$ makes it positive and grows fast for big gaps; $\sum$ collects all $n$ points into one total badness.

WHY squaring and not, say, absolute value?

1. Sign killed — $(-5{)}^2=25=(+5{)}^2$, above and below count equally, no cancellation.
2. Big misses hurt more — a gap of $10$ contributes $100$; a gap of $1$ contributes $1$. Outliers get shouted at.
3. Smooth — squares have derivatives everywhere, so calculus can find the bottom. Absolute value has a sharp corner calculus dislikes.

PICTURE. Each residual becomes a literal square; the area of every square is $e_i^2$; total inked area is RSS. Then, as we turn the two knobs, RSS traces out a smooth bowl — one lowest point. That lowest point is the best line.

Why does the bowl have exactly one bottom? RSS is a sum of squares of things linear in ${\beta }_0,{\beta }_1$ — that always makes an upward paraboloid. No fake valleys, no ambiguity.

---

Step 4 — Reading the bottom of the bowl with calculus

WHAT. At the lowest point of any smooth bowl, the ground is flat — the slope in every direction is zero. We have two knobs, so two flatness conditions:

$\frac{\partial \text{RSS}}{\partial {\beta }_0}=0,\frac{\partial \text{RSS}}{\partial {\beta }_1}=0$

The symbol $\partial$ (a "curly d") means: wiggle only this one knob, hold the other fixed, and ask how RSS changes. Setting it to $0$ says "wiggling this knob no longer lowers the cost — we're at the flat bottom for this direction."

WHY this tool? We want the minimum of a smooth function of two variables. The one universal signature of a minimum on smooth ground is zero slope in both directions. Derivatives are exactly the tool that measures slope. That is why calculus enters here and nothing else.

PICTURE. On the bowl, the bottom is where the tangent plane is perfectly horizontal — the two arrows (one per knob direction) both go flat.

---

Step 5 — The intercept condition: the line must pass through the centre

WHAT. Do the ${\beta }_0$ derivative. Using the chain rule on each squared term, $\frac{\partial }{\partial {\beta }_0}(y_i-{\beta }_0-{\beta }_1{x}_i{)}^2=2(y_i-{\beta }_0-{\beta }_1{x}_i)\cdot (-1)$:

$\frac{\partial \text{RSS}}{\partial {\beta }_0}=-2{\sum }_{i=1}^n(y_i-{\beta }_0-{\beta }_1{x}_i)=0⟹{\sum }_{i=1}^n{e}_i=0$

The $-2$ divides away; what's left says the residuals must add to zero. Positive gaps above balance negative gaps below.

Now expand $\sum (y_i-{\beta }_0-{\beta }_1{x}_i)=0$ and divide by $n$:

$\underset{\bar{y}}{\underbrace{\frac{1}{n}\sum y_i}}-{\beta }_0-{\beta }_1\underset{\bar{x}}{\underbrace{\frac{1}{n}\sum x_i}}=0⟹\boxed{{\beta }_0=\bar{y}-{\beta }_1\bar{x}}$

Here $\bar{x}$ and $\bar{y}$ (read "x-bar", "y-bar") are just the averages — the balance point of all the $x$'s and all the $y$'s.

WHY it matters. Rearranged, this equation is $\bar{y}={\beta }_0+{\beta }_1\bar{x}$ — the best line always passes through the centre of mass $(\bar{x},\bar{y})$ of the cloud. The intercept isn't free; once you pick a tilt ${\beta }_1$, the height is forced so the line pins the centroid.

PICTURE. The green centroid dot $(\bar{x},\bar{y})$; every best-fit line is a see-saw pivoting through it.

---

Step 6 — The slope condition: matching tilt to how x and y move together

WHAT. Do the ${\beta }_1$ derivative (same chain rule, but each term now also multiplied by $x_i$):

$\frac{\partial \text{RSS}}{\partial {\beta }_1}=-2{\sum }_{i=1}^n{x}_i(y_i-{\beta }_0-{\beta }_1{x}_i)=0$

Substitute the intercept we just found, ${\beta }_0=\bar{y}-{\beta }_1\bar{x}$, and the constants collapse into centred quantities:

${\sum }_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})-{\beta }_1{\sum }_{i=1}^n(x_i-\bar{x}{)}^2=0$

Solve:

$\boxed{{\beta }_1=\frac{{\sum }_{i=1}^n(x_i-\bar{x})(y_i-\bar{y})}{{\sum }_{i=1}^n(x_i-\bar{x}{)}^2}}$

Term by term — this is the heart, so read slowly. Recentre every point on the centroid: let $u_i=x_i-\bar{x}$ (how far right/left of centre) and $v_i=y_i-\bar{y}$ (how far up/down of centre).

• Numerator $\sum u_i{v}_i$ — for each point multiply "how far right" by "how far up." A point up-and-right ($+\cdot +$) or down-and-left ($-\cdot -$) gives a positive vote for an upward slope. A point down-and-right ($+\cdot -$) votes negative. This is the covariance — it measures do $x$ and $y$ rise together?
• Denominator $\sum u_i^2$ — how spread out the $x$'s are, the variance of $x$. Always positive.

So ${\beta }_1=\frac{\text{how much }x,y\text{ move together}}{\text{how much }x\text{ spreads}}$. It converts an agreement-of-movement into a per-unit tilt.

PICTURE. Each point coloured by the sign of its vote $u_i{v}_i$; the slope leans toward the majority vote, scaled down by the $x$-spread.

---

Step 7 — Edge cases: where the formula strains or breaks

WHAT & WHY. A trustworthy derivation must survive its own boundaries. Three cases:

(a) All $x$ identical — division by zero. If every house is the same size, $x_i=\bar{x}$ for all $i$, so every $u_i=0$ and the denominator $\sum u_i^2=0$. The formula divides by zero — undefined. Picture: all dots on one vertical line. There is no "rise per step right" because there is no step right. No slope can be defined; regression genuinely has no answer.

(b) No relationship — slope zero. If for every rightward point there's a mirror one that undoes its vote, the numerator $\sum u_i{v}_i=0$, so ${\beta }_1=0$. The best line is flat at $\bar{y}$: knowing $x$ tells you nothing, so predict the average. That's the right answer, not a failure.

(c) One outlier dragging the line. Because Step 3 squares errors, one far-off dot contributes a giant square and yanks the line toward itself. The formula still runs — it's the choice to square that gives outliers this power. (See 2.5-Assumptions-and-Diagnostics.)

PICTURE. Three little panels: vertical-line (undefined), symmetric cloud (flat), and one point hijacking the tilt.

---

The one-picture summary

Everything at once: the dots, the centroid the line pivots through (Step 5), the residual squares whose total we shrink (Step 3), and the final fitted line whose tilt is covariance-over-variance (Step 6).

Recall Feynman retelling — say it like a story

We had a scatter of dots and wanted one straight line to represent them. A line has two knobs: how high it sits (${\beta }_0$) and how steeply it tilts (${\beta }_1$). To score a line, we drop a vertical stick from each dot to the line — that's the miss, the residual. Adding raw misses is useless because ups cancel downs, so we square each miss into a little square of area, add all the areas, and call that total the cost. As we twist the two knobs, the cost traces a smooth bowl with exactly one bottom. Calculus finds that bottom by demanding the ground be flat in both knob-directions. The height knob's flatness says the misses must sum to zero — which forces the line through the dead centre $(\bar{x},\bar{y})$ of the cloud. The tilt knob's flatness says: lean the line by exactly how much $x$ and $y$ rise together, divided by how much $x$ spreads out. If $x$ never varies, there's no tilt to find (divide by zero). If $x$ and $y$ don't move together, the tilt is zero and you just guess the average. That's the whole of simple linear regression — two knobs, one bowl, one flat bottom.

Where this goes next: more than one feature → 2.2.02-Multiple-Linear-Regression; solving the bowl by walking downhill instead of algebra → 2.3-Gradient-Descent-for-Linear-Regression; when a straight line is the wrong shape → 4.2-Overfitting-vs-Underfitting; predicting yes/no instead of a number → 3.1-Logistic-Regression-Model.