Visual walkthrough — Multiple linear regression

Before line one, five plain-word promises about the letters you will meet:

• A feature is one measured number about a thing (a house's size, its number of bedrooms).
• We will have $n$ features in total — $n$ is just "how many measured numbers per house".
• The target $y$ is the number we want to predict (the house's price).
• A prediction $\hat{y}$ (read "y-hat") is our guess at $y$. The little hat means "estimated".
• Notation convention: a plain italic letter like $y$ is a single number. A bold letter is a bundle of numbers: a lower-case bold letter such as $\mathbf{y}$, $\mathbf{e}$, $\mathbf{\beta }$ is a vector (a stacked list of numbers), and an upper-case bold letter such as $\mathbf{X}$ is a matrix (a grid of numbers). Whenever you see bold, think "a whole collection at once".

That is the whole vocabulary. Let's go.

---

Step 1 — One prediction is a weighted sum

WHAT. We guess the price by giving each feature a weight and adding everything up: $\hat{y}=\underset{\text{starting value}}{\underbrace{{\beta }_0}}+\underset{\text{weight for }{x}_1}{\underbrace{{\beta }_1}}{x}_1+\underset{\text{weight for }{x}_2}{\underbrace{{\beta }_2}}{x}_2$

Read each symbol where it sits:

• ${\beta }_0$ (read "beta-zero") is the intercept — the guess before any feature speaks, a fixed offset.
• ${\beta }_1,{\beta }_2$ are weights — how many price-units one unit of that feature is worth.
• $x_1,x_2$ are the feature values for this one house. (Here we have $n=2$ features.)

WHY a weighted sum and not something fancier? Because "weighted sum" is the simplest rule where each feature contributes independently and proportionally. Double the size → double the size's contribution. That proportional behaviour is exactly what the word linear means.

PICTURE — Figure 1. The bars show each term as a stacked contribution building up to the final guess.

If you want curves instead of straight contributions, that is a different tool — see Polynomial Regression.

---

Step 2 — Stack every house into a table (a matrix)

WHAT. We have not one house but $m$ of them, each with $n$ features. Writing $m$ separate weighted-sum lines is painful. Instead we lay the data in a grid. A grid of numbers is called a matrix.

$\mathbf{X}=\left[\begin{array}{ccc}1& x_1^{(1)}& x_2^{(1)}\\ 1& x_1^{(2)}& x_2^{(2)}\\ ⋮& ⋮& ⋮\\ 1& x_1^{(m)}& x_2^{(m)}\end{array}\right],\mathbf{y}=\left[\begin{array}{c}{y}^{(1)}\\ y^{(2)}\\ ⋮\\ y^{(m)}\end{array}\right]$

Term by term:

• $m$ is the number of houses (rows); $n$ is the number of features (here $n=2$, so with the bias column $\mathbf{X}$ has $n+1$ columns).
• The superscript $(i)$ means "example number $i$" — a label, not a power. $x_1^{(2)}$ is house 2's size.
• One row of $\mathbf{X}$ = one house.
• One column of $\mathbf{X}$ = one feature across all houses.
• $\mathbf{y}$ is the tall stack of true prices, one per row.

WHY the leading column of 1's? The intercept ${\beta }_0$ has no feature attached. Give it a fake feature that is always $1$: then ${\beta }_0\cdot 1={\beta }_0$ folds neatly into the same sum. Now every weight has a column, and one clean rule handles all of them.

PICTURE — Figure 2. Rows are houses (colour-coded), columns are features; note the pale "always 1" column.

---

Step 3 — All predictions at once: $\hat{\mathbf{y}}=\mathbf{X}\mathbf{\beta }$

WHAT. Collect the weights into their own tall stack $\mathbf{\beta }=[{\beta }_0,{\beta }_1,{\beta }_2{]}^{\top }$ (the ${}^{\top }$ just means "written as a column"). Then matrix multiplication produces all predictions in one shot: $\hat{\mathbf{y}}=\mathbf{X}\mathbf{\beta }$

WHY does one multiplication give every prediction? Because matrix multiplication is defined as "take each row, multiply it entry-by-entry with the column, add up". That row-times-column operation is called a dot product — and a dot product of one house's row with $\mathbf{\beta }$ is exactly the weighted sum from Step 1: $\underset{\text{row }i}{\underbrace{[1,x_1^{(i)},x_2^{(i)}]}}\cdot \underset{\mathbf{\beta }}{\underbrace{\left[\begin{array}{c}{\beta }_0\\ {\beta }_1\\ {\beta }_2\end{array}\right]}}={\beta }_0+{\beta }_1{x}_1^{(i)}+{\beta }_2{x}_2^{(i)}={\hat{y}}^{(i)}$

So the tall output $\hat{\mathbf{y}}$ has one prediction per row — Step 1 repeated $m$ times, folded into one symbol.

PICTURE — Figure 3. Watch one row slide over the $\mathbf{\beta }$ column and collapse into one number.

The one-feature version of this is Simple Linear Regression — same idea, a single weight.

---

Step 4 — Measure how wrong we are (the cost)

WHAT. For a given $\mathbf{\beta }$, each house has an error = true minus predicted. Stack them: $\mathbf{e}=\mathbf{y}-\hat{\mathbf{y}}=\mathbf{y}-\mathbf{X}\mathbf{\beta }$ We square each error and add: this total is the sum of squared errors ${\mathbf{e}}^{\top }\mathbf{e}$. ${\mathbf{e}}^{\top }\mathbf{e}={\sum }_{i=1}^m(y^{(i)}-{\hat{y}}^{(i)}{)}^2$

Term by term:

• $\mathbf{e}$ is the vertical gap between the true dot and the model's plane, one gap per house.
• ${\mathbf{e}}^{\top }\mathbf{e}$ multiplies the error-column by itself → each error squared, then summed.

WHY square, and not just add the gaps? Two reasons. (1) Squaring makes every error positive, so gaps above and below the plane can't cancel out. (2) Squaring is smooth — it has a clean, single lowest point we can find with calculus. Absolute value has a sharp corner; the square does not.

PICTURE — Figure 4. Vertical error bars from each point to the plane; a small inset shows each bar squared.

---

Step 5 — The cost is a valley; slide to its bottom

WHAT. Let us give the total squared error a name: $J(\mathbf{\beta })$. Here the letter $J$ is simply our chosen symbol for the cost (also called the loss) — one number saying "how bad are these weights"; the $(\mathbf{\beta })$ reminds us the cost depends on which weights we picked. So $J(\mathbf{\beta })={\mathbf{e}}^{\top }\mathbf{e}.$ Think of $J(\mathbf{\beta })$ as a surface: pick any weights $\mathbf{\beta }$, get a height (the total squared error). Because of the squaring, this surface is a smooth bowl — a paraboloid (the 3-D cousin of a parabola) — with a lowest point.

WHY calculus / the gradient? The lowest point of a smooth bowl is where the surface is flat in every direction — its slope is zero. First, plain words for the tools:

• A partial derivative $\frac{\partial J}{\partial {\beta }_k}$ means "hold every weight fixed except ${\beta }_k$, wiggle only ${\beta }_k$ a tiny bit, and measure how fast $J$ changes". It is the slope of the bowl along one weight's axis. The curly $\partial$ (read "partial-dee") just flags that other weights are held still.
• The gradient $\frac{\partial J}{\partial \mathbf{\beta }}$ stacks all those one-axis slopes into a single vector: $[\frac{\partial J}{\partial {\beta }_0},\frac{\partial J}{\partial {\beta }_1},\dots {]}^{\top }$. It is "the slope in every direction at once", and it points uphill.

We use the gradient because zero-gradient pinpoints the bottom of the valley — no other tool locates a minimum this directly.

We need two slope rules; here is why each holds (both mimic ordinary one-variable calculus):

• Linear term. For a fixed vector $\mathbf{a}$, the quantity ${\mathbf{\beta }}^{\top }\mathbf{a}=a_0{\beta }_0+a_1{\beta }_1+\cdots$ has slope $a_k$ in direction ${\beta }_k$, so the whole gradient is just $\mathbf{a}$. That is the vector twin of $\frac{d}{d\beta }(a\beta )=a$. With $\mathbf{a}={\mathbf{X}}^{\top }\mathbf{y}$, the $-2{\mathbf{\beta }}^{\top }{\mathbf{X}}^{\top }\mathbf{y}$ term contributes $-2{\mathbf{X}}^{\top }\mathbf{y}$.
• Quadratic term. Write $\mathbf{A}={\mathbf{X}}^{\top }\mathbf{X}$. Then ${\mathbf{\beta }}^{\top }\mathbf{A}\mathbf{\beta }={\sum }_{j,k}{A}_{jk}{\beta }_j{\beta }_k$. Differentiating in ${\beta }_k$ hits ${\beta }_k$ once on the left (${\sum }_j{A}_{jk}{\beta }_j$) and once on the right (${\sum }_j{A}_{kj}{\beta }_j$); those two sums add. Because $\mathbf{A}={\mathbf{X}}^{\top }\mathbf{X}$ is symmetric ($A_{jk}=A_{kj}$, since $({\mathbf{X}}^{\top }\mathbf{X}{)}^{\top }={\mathbf{X}}^{\top }\mathbf{X}$), the two sums are equal and combine into $2\mathbf{A}\mathbf{\beta }$. That is the vector twin of $\frac{d}{d\beta }(A{\beta }^2)=2A\beta$ — the factor of 2 is exactly this "left copy plus right copy".

$\frac{\partial J}{\partial \mathbf{\beta }}=\underset{\text{from the }-2\text{ term}}{\underbrace{-2{\mathbf{X}}^{\top }\mathbf{y}}}+\underset{\text{from the quadratic term}}{\underbrace{2{\mathbf{X}}^{\top }\mathbf{X}\mathbf{\beta }}}$

PICTURE — Figure 5. The bowl, its flat bottom, and the downhill arrows (gradient) all pointing toward it.

If you cannot invert (next step), you literally walk downhill on this bowl instead — that is Gradient Descent. Same valley, different way down.

---

Step 6 — Flat slope ⟹ the Normal Equation

WHAT. Set the gradient to zero (bottom of the bowl) and tidy up: $-2{\mathbf{X}}^{\top }\mathbf{y}+2{\mathbf{X}}^{\top }\mathbf{X}\mathbf{\beta }=0⟹{\mathbf{X}}^{\top }\mathbf{X}\mathbf{\beta }={\mathbf{X}}^{\top }\mathbf{y}$ This last line is the Normal Equation. Reading each piece:

• ${\mathbf{X}}^{\top }\mathbf{X}$ — how features co-vary with each other (the "shape" of the bowl).
• ${\mathbf{X}}^{\top }\mathbf{y}$ — how each feature co-varies with the target (which way pulls down).

If ${\mathbf{X}}^{\top }\mathbf{X}$ can be inverted (undone), multiply both sides by its inverse: $\boxed{\mathbf{\beta }=({\mathbf{X}}^{\top }\mathbf{X}{)}^{-1}{\mathbf{X}}^{\top }\mathbf{y}}$

WHY is this a minimum and not a maximum? The matrix ${\mathbf{X}}^{\top }\mathbf{X}$ is always positive semi-definite — for any vector $\mathbf{v}$, ${\mathbf{v}}^{\top }{\mathbf{X}}^{\top }\mathbf{Xv}=\Vert \mathbf{Xv}{\Vert }^2\ge 0$ (a squared length can never be negative). So the bowl never curves downward; it can only open upward or, in flat directions, stay level. When $\mathbf{X}$ has full rank (its columns are independent), that inequality is strict ($>0$) — the matrix is then positive definite, the bowl is strictly curved up, and the single flat point is a genuine minimum. A bowl that only opens upward has no peak, so a maximum is impossible.

PICTURE — Figure 6. The flat tangent plane touching the bowl exactly at its lowest point.

---

Step 7 — The degenerate case: when the bowl has a flat trough

WHAT. The words "if invertible" and "full rank" in Step 6 hide a trap. Sometimes ${\mathbf{X}}^{\top }\mathbf{X}$ has no inverse — it is only positive semi-definite (some direction is perfectly flat). We call such a matrix singular. Then the formula cannot run.

WHY does this happen? Recall $n$ = number of features and $m$ = number of houses. Two everyday causes:

1. More unknowns than data ($m<n$): fewer houses than weights to pin down — infinitely many $\mathbf{\beta }$ fit equally well.
2. Perfect multicollinearity: one feature is an exact recipe of others, e.g. $x_3=2x_1+x_2$. The bowl then has a whole flat trough instead of one lowest point — you cannot name "the" bottom.

PICTURE — Figure 7. A normal bowl (one bottom) beside a trough-shaped bowl (a whole valley floor of ties).

---

Step 8 — Plug in real numbers (sanity check)

WHAT. To make everything concrete, here is a small self-contained table of three houses, each with a size (in square feet) and a bedroom count, plus its sold price (in thousands of dollars):

Size $x_1$ (sqft)	Bedrooms $x_2$	Price $y$ (k$)
1500	3	300
2000	4	400
1800	3	350

So the data bundles are $\mathbf{X}=\left[\begin{array}{ccc}1& 1500& 3\\ 1& 2000& 4\\ 1& 1800& 3\end{array}\right],\mathbf{y}=\left[\begin{array}{c}300\\ 400\\ 350\end{array}\right].$ With 3 houses and 3 unknowns (${\beta }_0,{\beta }_1,{\beta }_2$) the plane passes exactly through every point (zero error), and solving ${\mathbf{X}}^{\top }\mathbf{X}\mathbf{\beta }={\mathbf{X}}^{\top }\mathbf{y}$ gives $\mathbf{\beta }=\left[\begin{array}{c}{\beta }_0\\ {\beta }_1\\ {\beta }_2\end{array}\right]=\left[\begin{array}{c}-250\\ 0.1\\ 50\end{array}\right]\Rightarrow \hat{y}=-250+0.1x_1+50x_2.$

Check, house 1 (1500 sqft, 3 bed): $-250+0.1(1500)+50(3)=-250+150+150=300$ ✓ Check, house 2 (2000 sqft, 4 bed): $-250+0.1(2000)+50(4)=-250+200+200=400$ ✓ New house (1600 sqft, 3 bed): $-250+0.1(1600)+50(3)=-250+160+150=260$k.

WHY show this? To prove the abstract formula spits out ordinary, checkable prices — and that the plane truly threads all three data dots.

PICTURE — Figure 8. The fitted plane and the three data houses sitting exactly on it.

Because it hits every point exactly, this fit tells us nothing about new houses' error — that gap between "fits the data" and "predicts well" is Overfitting; you measure the good kind with R-squared and Adjusted R-squared.

---

The one-picture summary

PICTURE — Figure 9. The whole derivation compressed into one flow.

Data table → predictions by dot product → square the gaps → the cost is a bowl → its flat bottom is the Normal Equation. Read the whole page in one glance.

Recall Feynman retelling — say it like a friend

We want to guess a price from a few numbers, so we give each number a weight and add them up — that is one prediction. Do this for every house at once by stacking the numbers into a grid (a matrix) and multiplying — a single multiply does all the guessing. Now, our guesses will be wrong, so we measure the gap for each house, square it (so nothing cancels and the maths stays smooth), and add all the squares — that total is our badness score $J$. Picture that score as the height of a bowl: some weights sit high (bad), the best weights sit at the very bottom. The bottom is where the ground is flat, so we find where the slope (the gradient) is zero. Doing that gives one tidy equation, and if the data isn't secretly redundant we can solve it directly for the winning weights. If the data is redundant, the bowl flattens into a trough with no single bottom — then we nudge it back into shape (Ridge) or just roll downhill (gradient descent). That's the whole story.

Recall Quick self-test

Why the leading column of 1's in $\mathbf{X}$? ::: It gives the intercept ${\beta }_0$ a constant feature so it joins the same dot product. Why square the errors instead of summing them raw? ::: Squaring stops positive and negative gaps cancelling and keeps the cost smooth with one clear minimum. What is $J(\mathbf{\beta })$ and what shape does it have? ::: $J$ is the cost (total squared error); as a surface it is an upward-opening bowl (paraboloid) with a single lowest point when $\mathbf{X}$ is full rank. What does the gradient measure and where is it zero? ::: The gradient stacks the slope along each weight; it is zero at the flat bottom of the bowl — the least-squared-error weights. When is ${\mathbf{X}}^{\top }\mathbf{X}$ non-invertible? ::: When $m<n$ or when features are perfectly collinear — the bowl becomes a flat trough (only positive semi-definite).