Cost function (MSE) and gradient descent fitting: Foundations

This page assumes you know nothing. Before you can read the parent note, every squiggle in it must first become a picture in your head. We build them one at a time, each resting on the one before.

1. Dots on a grid — the data $(x_{i}, y_{i})$

Everything starts with data: a list of measurements. Say we recorded, for three houses, their size and their price. Each house is one row.

Picture: a flat grid. The horizontal axis is $x$ , the vertical axis is $y$ . Each example becomes one dot whose left-right position is $x_{i}$ and whose up-down position is $y_{i}$ .

Figure — Cost function (MSE) and gradient descent fitting

Why the topic needs this: the whole job is to draw a line through these dots. No dots, nothing to fit.

Recall

What does the subscript $i$ in $x_{i}$ mean? ::: It is a counter/label picking out which example — example number $i$ — not a mathematical operation.

2. How many dots — the number $n$

Picture: count the dots in the previous figure. There are 3. So $n = 3$ .

Why the topic needs this: later we average the errors. To average you must divide by the count, and that count is $n$ .

3. The Greek letters — $θ$ , $Σ$ , $α$

Mathematicians use Greek letters as short names. They are just names, like using a and b in code.

3a. What $θ_{0}$ and $θ_{1}$ actually do

A straight line needs exactly two pieces of information:

$θ_{0}$ — the height where the line crosses the vertical axis (the "intercept"). Slide the whole line up or down.
$θ_{1}$ — the tilt / steepness (the "slope"). How much $y$ rises for every 1 step right in $x$ .

For now, just picture these two knobs setting a line's height and tilt on the grid. In section 4 we give the line a proper name, $h_{θ} (x)$ , and write its equation. The full construction lives in 2.2.03-Hypothesis-representation-in-linear-regression.

3b. The subscript $j$ — which knob we mean

When we have several knobs $θ_{0}, θ_{1}, θ_{2}, \dots$ , we need a way to say "pick out one particular knob to talk about." That job goes to the letter $j$ .

3c. What $Σ$ means as a picture

$\sum_{i = 1}^{n} (something with i)$

Read left to right: "start the counter $i$ at $1$ (bottom), stop at $n$ (top), plug each value of $i$ into the expression, and add all the results into one number."

Why the topic needs it: the cost adds up the miss of every dot into one score, and $Σ$ is the compact way to write "add over all $n$ examples".

4. The prediction $h_{θ} (x)$ and the hat $\overset{y}{^}$

We also write $\overset{y}{^}_{i} = h_{θ} (x_{i})$ . The hat on $\overset{y}{^}$ ("y-hat") is a universal code for "this is an estimate, not the truth." So:

$y_{i}$ = the real value (a dot).
$\overset{y}{^}_{i}$ = the value the line gives at that same $x_{i}$ (a point on the line, directly above/below the dot).

Why the topic needs it: the gap between the dot $y_{i}$ and the line $\overset{y}{^}_{i}$ is the whole source of "wrongness". No hat, no way to name our guess.

Recall

What is the difference between $y_{i}$ and $\overset{y}{^}_{i}$ ? ::: $y_{i}$ is the true measured value (the dot); $\overset{y}{^}_{i}$ is the line's prediction at the same $x_{i}$ (a point on the line).

5. The error / residual $e_{i}$

Sign matters and covers all cases:

Line above the dot → $\overset{y}{^}_{i} > y_{i}$ → $e_{i}$ is positive.
Line below the dot → $\overset{y}{^}_{i} < y_{i}$ → $e_{i}$ is negative.
Line exactly through the dot → $\overset{y}{^}_{i} = y_{i}$ → $e_{i} = 0$ (perfect on that point).

Picture: the vertical red segments in figure s03 are the $e_{i}$ . Their length is how wrong we are; their direction (up/down) is the sign.

6. Squaring, and why " $^{2}$ "

The little raised $2$ means multiply the number by itself: $(- 3)^{2} = 9$ , $(4)^{2} = 16$ .

Two things squaring buys us, as a picture:

Everything becomes positive — a gap of $- 3$ and a gap of $+ 3$ both become $9$ , so they can't cancel.
Big misses hurt disproportionately — the curve $e^{2}$ is a smooth bowl (a parabola). Doubling the error quadruples the penalty.

Why a smooth bowl matters: a bowl has a single lowest point and a gentle slope everywhere, so we can "roll downhill" to the bottom. The absolute-value shape $∣ e ∣$ has a sharp kink at zero where "which way is downhill?" has no clean answer. Smoothness = we can use slopes (next).

7. The cost function $J (θ)$ — the MSE score

Now we can build the single number the whole topic tries to shrink. We already have, for each example: a residual $e_{i}$ , then its square $e_{i}^{2}$ . Add all those squares with $Σ$ , and divide by the number of examples to get an average — that average is the Mean Squared Error (MSE): literally the mean (average) of the squared errors.

Why the topic needs it: $J (θ)$ is the height of the landscape we are about to descend. Everything from here on is "make $J$ smaller."

Recall

In words, what does $J (θ)$ measure? ::: Half the average squared vertical miss of the line across all $n$ dots — one number saying how bad the current knob settings are.

8. Slope, derivative, and the gradient $\nabla J$

To "roll downhill" we need to measure steepness of that $J$ -landscape: at the spot where we're standing, does the cost $J$ go up or down if we tweak a knob a little, and by how much?

Picture: stand on the bowl surface. The slope is the tilt of the ground in the $θ_{j}$ direction. Positive slope = ground rises to the right = go left to descend. Negative slope = go right. Zero slope = you're at the flat bottom.

Why the topic needs it: the derivative $\frac{\partial J}{\partial θ _{j}}$ is the compass. Without it we'd have to guess blindly which way makes the line better.

9. The update arrow $:=$ and the step size $α$

$α$ (the step size) scales how far we move:

$α$ too big → we leap over the valley bottom and can bounce higher — divergence.
$α$ too small → correct direction but crawling — many steps.
$α$ just right → steady march to the bottom.

More on tuning it lives in 4.2.03-Learning-rate-schedules.

Prerequisite map

Each box is a symbol you now own; follow the arrows and you have rebuilt the whole parent note from zero.

Equipment checklist

Test yourself — say the answer aloud before revealing.

$x_{i}$ and $y_{i}$ ::: The input and the true output of example number $i$ — one dot on the grid.
$n$ ::: How many examples (dots) we have; we divide by it to average.
$θ_{0}$ ::: The line's height / intercept — where it crosses the vertical axis.
$θ_{1}$ ::: The line's tilt / slope — rise in $y$ per unit step in $x$ .
The index $j$ ::: A placeholder picking which knob — ranges over $0, 1$ (here two knobs, $θ_{0}$ and $θ_{1}$ ).
$h_{θ} (x)$ ::: The hypothesis — the line's predicted height at $x$ , equal to $θ_{0} + θ_{1} x$ ; our guess.
$\overset{y}{^}_{i}$ ::: The prediction for example $i$ (a point on the line), as opposed to the true $y_{i}$ .
$e_{i} = \overset{y}{^}_{i} - y_{i}$ ::: The signed vertical gap between line and dot — positive above, negative below, zero on it.
$\sum_{i = 1}^{n}$ ::: A loop that adds one term per example into a single total.
Squaring $e_{i}^{2}$ ::: Makes errors positive, punishes big misses more, and gives a smooth bowl we can descend.
$J (θ)$ ::: Half the Mean Squared Error — $\frac{1}{2 n} \sum e_{i}^{2}$ ; the landscape height we shrink (raw MSE is twice this).
$\frac{\partial J}{\partial θ _{j}}$ ::: The steepness of the cost $J$ in the direction of knob $θ_{j}$ — our downhill compass.
$\nabla J$ ::: The arrow of steepest uphill (all the slopes together); we step opposite to it.
$:=$ ::: "Overwrite this knob with a new value" — an instruction, not an equation.
$α$ ::: The step size; too big diverges, too small crawls.
Simultaneous update ::: Compute every slope on the OLD $θ$ values, then overwrite all knobs at once.

When every reveal feels obvious, you are ready for the main note and its neighbours 2.2.05-Normal-equation and 3.1.02-Convex-optimization.