Foundations — Normal equation closed-form solution

This page is the ground floor of the parent topic. Before we can appreciate why $\mathbf{\theta }=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^T\mathbf{y}$ is beautiful, we must earn every single symbol in it. We build from a single dot on paper up to the full matrix machine. If you can already read every line of the Equipment checklist at the bottom, skip ahead.

0. The scene: dots and a ruler

Imagine points scattered on graph paper. Each dot is one training example: something we measured (an input) paired with an outcome we want to predict (an output). Our whole job is to lay a straight ruler through them so it passes as close as possible to all the dots at once.

Look at the amber ruler above. The vertical cyan gaps between each dot and the ruler are the errors. "Best ruler" will mean: the ruler that makes those gaps, taken all together, as small as possible. Every symbol below exists to make that sentence precise.

1. A single number, a subscript, a superscript

Picture a spreadsheet. The subscript picks a column (which measured quantity), the superscript $(i)$ picks a row (which data point).

• Why the topic needs it: with $m$ examples and $n$ features, we cannot write out every value. Subscripts and superscripts let us name any cell with two little labels.

2. Input $\mathbf{x}$, output $y$, and how many of each

Picture: a table with $m$ rows and $n$ input columns, plus one final answer column $y$.

• Why the topic needs it: the parent's cost function sums ${\sum }_{i=1}^m$ — that "1 to m" is exactly "loop over every dot." You cannot read that sum without knowing $m$.

3. Vector — a list of numbers with a direction

Two pictures of the same object: an arrow from the origin (left), and a stack of numbers (right). Both are $\left[\begin{array}{c}3\\ 2\end{array}\right]$. The arrow view helps for geometry (length, angle, perpendicular); the stack view helps for computation.

• Why the topic needs it: $\mathbf{y}$ (all the true answers) and $\mathbf{\theta }$ (all the knobs) are vectors. Stacking them lets one formula act on the whole dataset at once.

4. $\mathbf{\theta }$ — the knobs we tune

Picture the ruler again: sliding ${\theta }_0$ up/down moves the whole ruler vertically; changing ${\theta }_1$ tilts it. Finding the best ruler = finding the best $\mathbf{\theta }$.

• Why the topic needs it: the normal equation's entire output is $\mathbf{\theta }$. It's the answer we solve for.

5. The prediction $h_{\mathbf{\theta }}(\mathbf{x})$ and the dot product

Picture: the dot product slides one vector along the other, multiplying overlapping entries — a single "weighted total." That total is the height of the ruler above that input point.

• Why $x_0=1$? Notice ${\theta }_0$ needs a partner $x_0$ so the pattern stays uniform. We invent a fake feature $x_0=1$ for every example. Then ${\theta }_0{x}_0={\theta }_0$, giving us the intercept for free while keeping one clean formula.

6. The error and the cost function $J(\mathbf{\theta })$

• The $\frac{1}{2m}$ is cosmetic: $\frac{1}{m}$ averages over dataset size, and the $\frac{1}{2}$ will cancel a factor of 2 when we differentiate. Neither changes where the minimum is.
• Why the topic needs it: the normal equation is the exact place where $J$ bottoms out. $J$ is the thing we minimize.

7. Matrices, transpose, and the design matrix $\mathbf{X}$

The figure shows the shapes clicking together. Multiplying $\mathbf{X}$ ($m\times (n+1)$) by $\mathbf{\theta }$ ($(n+1)\times 1$) gives $\mathbf{X}\mathbf{\theta }$: a single column of $m$ predictions — one for every dot, all at once. Matrix multiply is just "do the dot product of each row with the vector."

8. Gradient $\nabla$, the minimum, and "set to zero"

Because we squared the errors, $J$ plotted against $\mathbf{\theta }$ is a bowl (a paraboloid) — the amber surface above. At the very bottom of a smooth bowl the ground is perfectly flat, so every slope is zero. That is why we write: ${\nabla }_{\mathbf{\theta }}J=0$

• "Set the gradient to zero" means "find the flat spot," and for a bowl there's exactly one flat spot: the minimum. Solving that flatness condition is the normal equation.
• Contrast: Gradient Descent walks downhill along $-\nabla J$ in many small steps; the normal equation jumps straight to where $\nabla J=0$ in one algebra move.

9. Inverse, singular, and rank

Prerequisite map

Every arrow says "you need the left box before the right box makes sense." The final box $K$ is the parent topic. Related roads out: Ridge Regression (fixes singular cases by adding to the diagonal), QR Decomposition (a numerically safer way to solve the same system), Feature Scaling and Polynomial Regression (how you build $\mathbf{X}$), and Linear Regression Fundamentals for the big picture.

Equipment checklist

Cover the right side; can you answer each before revealing?