Foundations — Ordinary least squares derivation

This page is the ground floor. The parent note throws a lot of notation at you very fast: $x_i$, ${\theta }^T{x}_i$, $\Vert X\theta -y{\Vert }^2$, $X^{T}X$, gradients, transposes. If any of those looked like hieroglyphs, start here. We build every single symbol from a picture before the parent is allowed to use it.

0. The scene, before any symbols

Picture a scatter of dots on graph paper. Horizontal axis = something we know (hours studied). Vertical axis = something we want to predict (exam score). Each dot is one student.

Our whole job is to draw one line through this cloud. Then, given a new student's study hours, we read the height of the line to guess their score. OLS is just a precise rule for choosing which line — the one whose total up-and-down miss from the dots is as small as possible.

Everything below is the vocabulary needed to say that sentence in mathematics.

1. A data point: $x_i$, $y_i$, and the subscript $i$

Why does the topic need this? Because we will have to sum over all dots, and to write "add up something for every dot" cleanly we need a way to say "the $i$-th one".

2. How many dots, how many measurements: $m$ and $n$

If we only know "hours studied", then $n=1$. If we also know "hours slept" and "attendance percent", then $n=3$. Each feature is one more axis of the room the dots live in.

3. The vector $x_i$: a dot with several coordinates, and ${\mathbb{R}}^n$

When there is more than one feature, a single dot's input is not one number but a list of numbers — one per feature. We call an ordered list of numbers a vector.

So $x_{i2}$ has two subscripts: first says which dot, second says which feature. Read it as "$x$, dot $i$, feature $2$".

Why the topic needs this: real problems have many features, and writing $x_i$ as one vector lets us handle 1 feature or 10,000 features with the same symbol.

4. Parameters $\theta$ and the intercept ${\theta }_0$

5. Superscript $T$: the transpose, and the dot product ${\theta }^T{x}_i$

The parent writes the prediction as ${\hat{y}}_i={\theta }^T{x}_i$. Two new pieces of notation: the little $T$, and the hat.

5a. Transpose $T$ — turn a column into a row

$\theta =\left[\begin{array}{c}{\theta }_0\\ {\theta }_1\\ {\theta }_2\end{array}\right]{\theta }^T=\left[\begin{array}{ccc}{\theta }_0& {\theta }_1& {\theta }_2\end{array}\right]$

Why bother? Because to multiply a row by a column in the matrix rules, the shapes have to line up: row-vector times column-vector gives a single number.

5b. The product ${\theta }^T{x}_i$ — multiply matching entries and add

This one number is the prediction: "start at the intercept, then add each slope times its feature". This operation — multiply-matching-and-sum — is called the dot product.

6. The prediction hat: $\hat{y}$

So there are two heights above each dot's horizontal position: the real dot ($y_i$) and the point on our line directly above/below it (${\hat{y}}_i={\theta }^T{x}_i$). The gap between them is the whole story of OLS.

7. The residual and why we square it

Why not absolute value $|r_i|$, which is also always positive? Because the squaring gives a smooth bowl-shaped cost with a clean derivative — you'll see in the parent that $x^2$ differentiates to $2x$, which makes the calculus solvable in one step. Absolute value has a sharp corner and no clean formula.

8. The summation sign $\sum$ and the cost $J(\theta )$

Now the parent's cost function reads in plain English:

9. Stacking into matrices: $X$, $y$, and $X\theta$

Writing a separate line per dot is tedious. Linear algebra lets us bundle all $m$ dots into single objects.

Why the column of 1s? Because ${\theta }^T{x}_i={\theta }_0\cdot 1+{\theta }_1{x}_{i1}+\cdots$. If the first feature is always $1$, then ${\theta }_0$ automatically becomes the intercept — no special-case code. That single trick lets us write all $m$ predictions at once:

10. The norm $\Vert \cdot {\Vert }^2$: total squared length

Set $v=X\theta -y$ (the full column of residuals). Then $\Vert X\theta -y{\Vert }^2$ is exactly the sum of squared residuals from Section 8 — just written compactly. So the entire cost collapses to:

$J(\theta )=\frac{1}{2m}\Vert X\theta -y{\Vert }^2=\frac{1}{2m}(X\theta -y{)}^T(X\theta -y).$

Every symbol in that line now has a picture behind it: $X$ is the grid of dots, $\theta$ the knobs, $X\theta$ the predictions, $X\theta -y$ the residual arrows, $\Vert \cdot {\Vert }^2$ the total squared length of those arrows.

11. What "gradient = 0" will mean (a preview)

The parent finds the bottom of the bowl by setting the gradient to zero. Full detail lives in the derivation, but here's the foundation:

That single condition, written out with all the matrix rules, becomes the parent's normal equation $X^{T}X\theta =X^{T}y$. You are now equipped to read that derivation symbol by symbol.

Prerequisite map

Equipment checklist

Test yourself — say each answer out loud before revealing.

Prerequisites to revisit if any answer stalled: Linear Regression Fundamentals, Feature Scaling, and for what comes next Gradient Descent, Pseudoinverse, Ridge Regression, Maximum Likelihood Estimation.