Multiple linear regression: Foundations — AI-ML deep dive

This page assumes you have seen nothing. We will meet every symbol that appears on the parent note one at a time, draw what it means, and only then use it. If you can add and multiply, you can finish this page.

1. A "feature" and a "target" — the raw ingredients

Picture a house. We can write down facts about it — size, bedrooms, age. Each fact is a feature. The thing we don't know yet, the price, is the target.

Notation for features. We write the features as $x_{1}, x_{2}, \dots, x_{n}$ .

The letter $x$ means "a feature value".
The little number below (the subscript) says which feature: $x_{1}$ is size, $x_{2}$ is bedrooms, and so on.
$n$ is simply how many features we have. If we track size, bedrooms, age, then $n = 3$ .

The target is written $y$ .

2. Many houses at once — the superscript

We rarely have one house. We have a whole list. To point at a specific house we add a superscript in brackets:

$x_{2}^{(3)} = feature 2 (bedrooms) of house number 3$

The letter $m$ means how many examples (houses) we have. So we have $m$ houses, each described by $n$ features.

Recall

What does the little number below $x$ (subscript) tell you? ::: Which feature it is (which column). What does the bracketed number above $x$ tell you? ::: Which example/row it is (which house). If we have 100 houses and track size, bedrooms, age, what are $m$ and $n$ ? ::: $m = 100$ , $n = 3$ .

3. The prediction formula — weighted addition, spelled out

Here is the parent note's central line:

$\overset{y}{^} = β_{0} + β_{1} x_{1} + β_{2} x_{2} + \dots + β_{n} x_{n}$

Let us earn every symbol.

So the formula reads, in plain English:

**prediction = baseline + (weight₁ × feature₁) + (weight₂ × feature₂) + … **

That's it. It's weighted addition. The picture below shows each feature contributing a slice.

Recall

In $\overset{y}{^} = - 150 + 0.2 x_{1} + 50 x_{2}$ , what is the intercept? ::: $- 150$ . What does $β_{1} = 0.2$ mean in words? ::: Each extra square foot adds $0.2k to the predicted price, holding bedrooms fixed.

4. From "a line" to "a hyperplane"

With one feature, $\overset{y}{^} = β_{0} + β_{1} x_{1}$ is a straight line on a 2D graph. With two features it becomes a flat plane floating above the $(x_{1}, x_{2})$ floor. With more features we can't draw it, but it's the same flat shape in more dimensions — mathematicians call any such flat sheet a hyperplane.

5. Vectors and matrices — a filing cabinet for numbers

Writing the formula out for every house is exhausting. Instead we stack numbers into grids.

The parent's data matrix looks like:

$X = 11 ⋮ 1 x_{1}^{(1)} x_{1}^{(2)} ⋮ x_{1}^{(m)} \dots \dots ⋱ \dots x_{n}^{(1)} x_{n}^{(2)} ⋮ x_{n}^{(m)}$

Now we can name the whole bundle of weights as one object.

6. The dot product — one prediction in one stroke

To predict for one house, take its row of features and its matching weights, multiply matching pairs, and add. That operation is the dot product.

From dot product to full matrix multiplication. Multiplying a matrix by a vector is just "do this dot product once per row":

So $\hat{y} = X β$ produces one predicted number per house, all at once. One symbol replaces $m$ separate formulas.

Recall

Row $(1, 1600, 3)$ dotted with weights $(- 150, 0.2, 50)$ — what do you get? ::: $- 150 + 0.2 \cdot 1600 + 50 \cdot 3 = 320$ . An $m \times (n + 1)$ matrix times an $(n + 1) \times 1$ vector has what shape? ::: $m \times 1$ — one prediction per row. What does $X_{i 0}$ always equal, and why? ::: $1$ — it is the 0-th column, the invented constant feature that carries the intercept.

7. The transpose — flipping a grid on its side

The parent note writes $X^{⊤}$ (read "X transpose").

8. Error, squaring, and "best fit"

Our guess $\overset{y}{^}$ won't match the true $y$ perfectly. For house $i$ , the gap is the error (or residual), written $e^{(i)}$ :

$e^{(i)} = y^{(i)} - \overset{y}{^}^{(i)}$

$J (β) = \frac{1}{m} \sum_{i = 1}^{m} (e^{(i)})^{2} = \frac{1}{m} \sum_{i = 1}^{m} (y^{(i)} - \overset{y}{^}^{(i)})^{2}$

The big $\sum$ (Greek capital sigma) just means "add these up for every house, from house $1$ to house $m$ ". The $\frac{1}{m}$ turns the total into an average. We write $J (β)$ because the score depends on which weights $β$ we pick.

9. The matrix inverse — "undoing" a multiplication

Before we can read the Normal Equation we need one last symbol: the little $^{- 1}$ .

Common mistake When

$X^{⊤} X$ has NO inverse When does the formula break? $X^{⊤} X$ is singular (has no inverse) in two everyday situations:

Too few houses: fewer examples than weights, $m < n + 1$ . There isn't enough data to pin down every weight, so infinitely many weight sets fit equally — the formula can't pick one.
Redundant features (collinearity): one feature is an exact recipe of others, e.g. $x_{3} = 2 x_{1} + x_{2}$ , or "size in sq ft" and "size in sq m" as two columns. The duplicate carries no new information, so a column of $X$ is a copy of others — again no unique answer.

What to do instead:

Drop the redundant/duplicated feature (inspect the correlation matrix).
Add a small nudge to the diagonal so it becomes invertible again — that is exactly Ridge Regression.
Skip the inverse entirely and walk downhill with Gradient Descent, which still works when $X^{⊤} X$ is singular.

Having many redundant or noisy features also invites Overfitting, a separate danger covered later.

10. How it all connects

Each box on the left is something we built from zero on this page; together they feed the topic in the final box. Later ideas — Ridge Regression for when the algebra breaks, R-squared and Adjusted R-squared for scoring the fit, and Overfitting for its dangers — all sit on top of these foundations.

Equipment checklist

Test yourself — reveal only after answering.

What does

\overset{y}{^}

mean and how is it different from

y

\overset{y}{^}

is the model's guess;

y

is the true value from data.

What is a feature? A target?

A feature is one measured fact about a thing; the target is the number we want to predict.

x_{3}^{(7)}

, what do the

3

and the

(7)

each mean?

3

= which feature (column);

(7)

= which example/house (row).

What does

x_{0}^{(i)}

equal, and what is it for?

It equals

1

for every house; it is the constant feature that carries the intercept

β_{0}

What do

β_{0}

and

β_{i}

represent?

β_{0}

is the baseline (intercept); each

β_{i}

is the weight for feature

i

What is the bold

β

, and how many entries does it have?

The column vector stacking all weights; it has

n + 1

entries (intercept plus one per feature).

What is the shape of the data matrix

X

m \times (n + 1)

—

m

rows (houses),

n + 1

columns (features plus the 1's column).

What does the entry

X_{ik}

mean?

The number in row

i

, column

k

X

Why is there a column of 1's in

X

So the intercept

β_{0}

becomes just another weight in the dot product.

What is a dot product in one sentence?

Multiply matching pairs of numbers, then add them all.

When can two matrices be multiplied, and what shape results?

When the touching sizes match (

p \times q

times

q \times r

); result is

p \times r

What does the transpose

X^{⊤}

do?

Flips rows into columns and columns into rows.

What does the inverse

A^{- 1}

do, and what is

A^{- 1} A

It undoes

A

(matrix "division");

A^{- 1} A = I

Name two situations where

X^{⊤} X

has no inverse.

Fewer examples than weights (

m < n + 1

), or redundant/collinear features.

Why do we square the errors before adding them?

So positives and negatives don't cancel, and big misses are punished more.

What does

\sum_{i = 1}^{m}

instruct you to do?

Add the following quantity for every example from

1

m

What does "best fit" mean numerically?

The weights

β

that make the MSE cost

J

as small as possible.

Write the Normal Equation from memory.

β = (X^{⊤} X)^{- 1} X^{⊤} y