Cost function (MSE) and gradient descent fitting: Visual walkthrough

This page rebuilds the central result of the parent note: a recipe that keeps nudging a line until it fits a cloud of dots as tightly as possible. We will build it one symbol at a time. We assume you know nothing except that we have some dots on a graph and want a line through them, and — this is a promise — every symbol is earned before it is used. By the end of Step 5 you will have assembled, piece by piece, the boxed rule that runs the whole method.

Step 0 — What are we even looking at?

Figure — Cost function (MSE) and gradient descent fitting

$x_{i}$ ::: the input of the $i$ -th example (a dot's horizontal position in the data world)
$y_{i}$ ::: the true output of the $i$ -th example (the dot's vertical position)
$n$ ::: the number of training examples — how many dots we have (so $i$ runs $1, 2, \dots, n$ )
$θ_{0}$ ::: the intercept — where the line crosses the $y$ -axis (a knob, lives in parameter world)
$θ_{1}$ ::: the slope — how steep the line tilts (the other knob)

Everything that follows is a machine for rolling downhill in the right-hand picture.

Step 1 — Draw one prediction and its error

WHAT. Pick a single line (fix $θ_{0}, θ_{1}$ ). For one dot $x_{i}$ , the line predicts a height. Call it $h_{θ} (x_{i})$ — read it as "the hypothesis, tuned by $θ$ , evaluated at $x_{i}$ ."

$h_{θ} (x_{i}) = θ_{0} + θ_{1} x_{i}$

$θ_{0}$ ::: the starting height at $x = 0$
$θ_{1} x_{i}$ ::: how much we rise by the time we reach $x_{i}$ (slope × distance travelled)
$h_{θ} (x_{i})$ ::: the total predicted height, the point on the line above $x_{i}$

WHY. Before we can measure "wrongness" we need the prediction and the truth side by side. The vertical gap between them is the whole game.

PICTURE. The red dashed segment below is the error — the signed vertical distance from the line up (or down) to the true dot.

Step 2 — Turn all the errors into ONE number: the cost

WHAT. We have $n$ residuals $e_{1}, \dots, e_{n}$ (one per dot). We compress them into a single "how bad is this line?" number by squaring each and averaging.

$J (θ) = \frac{1}{2 n} \sum_{i = 1}^{n} e_{i}^{2} = \frac{1}{2 n} \sum_{i = 1}^{n} (h_{θ} (x_{i}) - y_{i})^{2}$

$e_{i}^{2}$ ::: area of a square built on the error segment — always $\geq 0$ , and it grows fast for big errors
$\sum_{i = 1}^{n}$ ::: add up those square-areas over every dot, from the first ( $i = 1$ ) to the last ( $i = n$ )
$\frac{1}{n}$ ::: divide by the dot-count $n$ to get an average, so more data doesn't automatically mean bigger cost
$\frac{1}{2}$ ::: a bookkeeping fraction chosen now to cancel a "2" that appears later (Step 4)

WHY squares and not, say, $∣ e_{i} ∣$ ? Two reasons, both visible:

Squaring erases the sign so above-errors and below-errors can't cancel to a false "zero cost."
A square has no sharp corner at the bottom — it is smooth — so later we can measure its slope with calculus. An absolute value has a kink at zero and no well-defined slope there.

PICTURE. Literally draw a square on each error. Cost = (half) the average shaded area.

Step 3 — The cost is a bowl over parameter world

WHAT. Fix the data. Now let the two knobs $θ_{0}, θ_{1}$ vary. Every choice of $(θ_{0}, θ_{1})$ gives one line, hence one cost $J$ . Plot that height above the $(θ_{0}, θ_{1})$ floor.

WHY this matters. Because $J$ is a sum of squares of things linear in the $θ$ 's, the surface is a smooth, upward-opening bowl — one single lowest point, no fake valleys to get stuck in (this is the convexity that makes the whole method safe). "Best line" now means "the bottom of the bowl."

PICTURE. The bowl, its unique minimum $⋆$ , and a marble sitting somewhere up the side — that marble is our current guess.

Step 4 — Which way is downhill? Ask the derivative

WHAT. To roll the marble down we need the local downhill direction. In one dimension the tool that reports "which way does the height change, and how fast?" is the derivative — the slope of the curve at a point. In two dimensions we ask it once per knob. We use the index $j$ as a stand-in for "which knob," so here $j$ takes just two values: $j = 0$ (the intercept) and $j = 1$ (the slope). The partial derivative $\frac{\partial J}{\partial θ _{j}}$ freezes every knob except $θ_{j}$ and measures the slope along that one axis.

WHY the derivative and not something else? We want the steepest local direction, and "steepness of a surface" is exactly what derivatives were invented to measure. No other tool gives a direction that's guaranteed to reduce $J$ for a small step.

Now compute it. Chain rule: the derivative of (something) $^{2}$ is $2 \cdot (something) \cdot (its derivative)$ .

= \frac{1}{2n}\sum_{i=1}^{n} \underbrace{2\big(h_\theta(x_i)-y_i\big)}_{\text{from squaring}} \cdot \underbrace{\frac{\partial h_\theta(x_i)}{\partial \theta_j}}_{\text{inside derivative}}$$ The "$2$" meets the "$\frac{1}{2}$" and both vanish — that's the whole reason the $\frac{1}{2}$ was planted in Step 2: $$\frac{\partial J}{\partial \theta_j} = \frac{1}{n}\sum_{i=1}^{n}\big(h_\theta(x_i)-y_i\big)\cdot\frac{\partial h_\theta(x_i)}{\partial \theta_j}$$ The last piece, $\frac{\partial h_\theta(x_i)}{\partial \theta_j}$, asks: *if I wiggle knob $j$ a tiny bit, how much does this one prediction move?* Recall $h_\theta(x_i)=\theta_0+\theta_1 x_i$, and to take $\partial/\partial\theta_j$ we hold the other knob still and treat it as a constant. - **For $j=0$ (intercept):** in $\theta_0+\theta_1 x_i$ the $\theta_1 x_i$ part is frozen (a constant), and $\theta_0$ appears just once, plainly. Bumping $\theta_0$ up by $1$ lifts the prediction by exactly $1$ — so the derivative is $1$. - **For $j=1$ (slope):** now $\theta_0$ is the frozen constant, and $\theta_1$ is multiplied by $x_i$. Bumping $\theta_1$ up by $1$ lifts the prediction by $x_i$ (the further out the dot, the bigger the swing) — so the derivative is $x_i$. $$\frac{\partial h_\theta(x_i)}{\partial \theta_0}=1, \qquad \frac{\partial h_\theta(x_i)}{\partial \theta_1}=x_i.$$ To write both cases in one clean symbol, define $x_i^{(j)}$ = "the multiplier that knob $j$ is attached to in the prediction": $x_i^{(0)}=1$ (the intercept multiplies nothing, i.e. a hidden $1$) and $x_i^{(1)}=x_i$. Then $\frac{\partial h_\theta(x_i)}{\partial \theta_j}=x_i^{(j)}$. - Turning $\theta_0$ shifts the whole line up by $1$ per unit — its "leverage" on the prediction is $1$. - Turning $\theta_1$ tilts the line; the effect at $x_i$ scales with $x_i$ — far-out points feel it more. **PICTURE.** A slice of the bowl along the $\theta_1$ axis: a parabola. The tangent line's tilt **is** the derivative. Positive tilt ⇒ we're on the right wall ⇒ go left; negative tilt ⇒ left wall ⇒ go right. ![[deepdives/dd-ai-ml-2.2.04-d2-s05.png]] > [!formula] The gradient (both knobs at once) > $$\frac{\partial J}{\partial \theta_j}=\frac{1}{n}\sum_{i=1}^{n}\big(h_\theta(x_i)-y_i\big)\,x_i^{(j)},\qquad j\in\{0,1\}$$ > with $x_i^{(0)}=1$ (the bias' leverage) and $x_i^{(1)}=x_i$. Each term = **error × leverage**. --- ## Step 5 — Take a step: the update rule **WHAT.** We know the uphill slope for each knob. To go **down**, subtract a small multiple of it. Every symbol below has now been earned: $\theta_j$ and $j\in\{0,1\}$ (Step 4), $\alpha$ (defined just below), $n$ (Step 0), $h_\theta(x_i)-y_i$ (Step 1), and $x_i^{(j)}$ (Step 4). $$\boxed{\;\theta_j := \theta_j - \alpha\,\frac{1}{n}\sum_{i=1}^{n}\big(h_\theta(x_i)-y_i\big)\,x_i^{(j)}\;}\qquad j\in\{0,1\}$$ - $:=$ ::: "replace the old value with this new one" - $\alpha$ ::: the ==learning rate== — how far along the downhill direction we step - the minus sign ::: turns the *uphill* gradient into a *downhill* move **WHY the minus works in every case:** - Slope **positive** (bowl rising to the right) → subtract a positive → $\theta_j$ moves **left**, toward the bottom. ✓ - Slope **negative** (bowl rising to the left) → subtract a negative → $\theta_j$ moves **right**, toward the bottom. ✓ - Slope **exactly zero** (we're at the bottom) → subtract $0$ → we **stop**. This is convergence, not a bug. ✓ **PICTURE.** One marble hop: current $\theta$, arrow $-\alpha\nabla J$, landing spot lower on the bowl. ![[deepdives/dd-ai-ml-2.2.04-d2-s06.png]] > [!mistake] Update simultaneously > Compute **both** new values from the **old** $\theta_0,\theta_1$, then overwrite. If you update $\theta_0$ first and feed it into $\theta_1$'s gradient, you're descending a distorted landscape. --- ## Step 6 — The one knob that changes everything: $\alpha$ **WHAT / WHY.** $\alpha$ only scales the step length; but its size decides *whether we ever arrive*. - $\alpha$ **too small** → each hop is a crawl → correct but painfully slow. - $\alpha$ **just right** → smooth glide into the minimum. - $\alpha$ **too big** → we leap *past* the bottom onto the opposite wall, higher than before → cost **grows** and we diverge. **PICTURE.** Three marbles on the same parabola with three step sizes — crawl, glide, blow-up. ![[deepdives/dd-ai-ml-2.2.04-d2-s07.png]] If features live on wildly different scales the bowl becomes a stretched canyon and one $\alpha$ can't suit both axes — that's why we do [[2.06-Feature-scaling-and-normalization|feature scaling]], and why practitioners use [[4.2.03-Learning-rate-schedules|learning-rate schedules]]. Summing over **all** $n$ points every step is *batch* descent; sampling one point per step is [[4.2.01-Stochastic-gradient-descent|stochastic gradient descent]]. The exact bottom can also be jumped to in closed form via the [[2.2.05-Normal-equation|normal equation]]. --- ## Step 7 — Watch one real step (numbers) **WHAT.** Data $(1,2),(2,4),(3,5)$, so $n=3$, start $\theta_0=\theta_1=0$, $\alpha=0.1$. With a flat line at height $0$ the predictions are all $0$, so the errors are $e_i=0-y_i=(-2,-4,-5)$. $$J(\theta)=\frac{1}{2\cdot 3}\big[(-2)^2+(-4)^2+(-5)^2\big]=\frac{4+16+25}{6}=\frac{45}{6}=7.5$$ $$\frac{\partial J}{\partial\theta_0}=\tfrac13\big[(-2)+(-4)+(-5)\big]=-\tfrac{11}{3}\approx-3.667,\qquad \frac{\partial J}{\partial\theta_1}=\tfrac13\big[(-2)(1)+(-4)(2)+(-5)(3)\big]=-\tfrac{25}{3}\approx-8.333$$ $$\theta_0:=0-0.1(-3.667)=0.367,\qquad \theta_1:=0-0.1(-8.333)=0.833$$ **PICTURE.** The flat starting line lifts and tilts toward the dots after one hop. ![[deepdives/dd-ai-ml-2.2.04-d2-s08.png]] Both gradients were negative (line below all dots) → both knobs increased → the line rose and steepened, exactly as your eye expects. --- ## The one-picture summary ![[deepdives/dd-ai-ml-2.2.04-d2-s09.png]] Left = data world: dots, line, error segments. Right = parameter world: the bowl and the marble's downhill path. The arrow between them is the update rule, ferrying "how wrong the line is" into "how to nudge the knobs." > [!recall]- Feynman retelling — say it back in plain words > We have $n$ dots and want a line. For a given line, the **error** at each dot is the vertical gap from line to dot. We **square** each gap (so signs can't cancel and big misses hurt more, and because squares are smooth), average them, and call that single number the **cost**. Now imagine every possible line as a spot on a floor whose coordinates are the line's intercept and slope; stack the cost as height, and you get a **bowl** with one lowest point — the best line. Standing somewhere on the bowl, the **derivative** tells us which way is uphill and how steep. We step the opposite way by a distance $\alpha$: new knob = old knob − $\alpha$ × slope. If the slope is zero we're already at the bottom and stop on our own. Repeat, and the marble slides to the best line. Too big an $\alpha$ overshoots and climbs the far wall; too small and it crawls. Each gradient term is literally **error × how much that knob moves the prediction** — that's the whole rule. > [!recall]- Quick checks > What does the minus sign in the update do? ::: Converts the uphill gradient into a downhill step. > Why the $\frac{1}{2}$ in the cost? ::: It cancels the $2$ that the power rule drops when differentiating the square. > What happens at the minimum? ::: The gradient is $0$, so the update subtracts nothing and we stop. > Why square instead of absolute value? ::: Squares are smooth (differentiable) everywhere and keep positive/negative errors from cancelling. > What does $n$ stand for, and over what does $j$ range? ::: $n$ is the number of training examples (dots); $j$ ranges over $\{0,1\}$, the two knobs (intercept and slope). > What is $x_i^{(0)}$ and why is it $1$? ::: The intercept's hidden multiplier; bumping $\theta_0$ shifts every prediction by exactly $1$, so its leverage is $1$.