Worked examples — Cost function (MSE) and gradient descent fitting

This page is a case atlas. The parent note built the two machines: the cost $J(\theta )$ (how wrong we are) and the update rule (how we fix it). Here we run those machines by hand across every situation they can meet — good learning rates, bad ones, flat data, single points, and a real word problem.

Recall the two tools we reuse everywhere (from the parent). First the cost — our single number for "how wrong is this line?":

And the update lines — how we push the line downhill:

The word residual just means $e_i$ — the leftover gap between what we said and what happened. Positive residual = we guessed too high; negative = too low.

---

The scenario matrix

Every gradient-descent problem lands in one of these cells. The examples below are labelled with the cell they cover.

Cell	What makes it special	Covered by
A. Baseline good $\alpha$	Small clean dataset, sane step size, cost drops	Ex 1
B. $\alpha$ too large (divergence)	Overshoot; cost grows each step	Ex 2
C. $\alpha$ too small	Cost drops but crawls; many steps needed	Ex 3
D. Perfect fit / zero residual	Data lies exactly on a line; gradient $=0$	Ex 4
E. Flat / degenerate data ($x$ all equal)	${\theta }_1$ can't be determined; only ${\theta }_0$ moves usefully	Ex 5
F. Negative slope data	Best line goes down; check signs of the gradient	Ex 6
G. Single point ($n=1$)	Degenerate limit; infinitely many perfect lines	Ex 7
H. Real-world word problem	Units, interpretation, prediction from fitted line	Ex 8
I. Exam twist: start at the optimum	Already minimised; verify gradient stays zero	Ex 9

---

Ex 1 — Cell A: baseline, a healthy learning rate

Forecast: The true line is obviously $y=x$ (so ${\theta }_1$ should climb toward $1$, ${\theta }_0$ stay near $0$). Guess: after one step ${\theta }_1$ jumps up a lot, ${\theta }_0$ a little.

1. Predictions. $h(1)=h(2)=h(3)=0$. Why this step? Both parameters are zero, so the line is flat on the $x$-axis — every prediction is $0$.
2. Residuals $e_i=h(x_i)-y_i$: $e_1=-1,e_2=-2,e_3=-3$. Why this step? These signed gaps drive both updates; all negative means we're guessing too low everywhere.
3. Gradients. $\frac{\partial J}{\partial {\theta }_0}=\frac{1}{3}(-1-2-3)=-2,\frac{\partial J}{\partial {\theta }_1}=\frac{1}{3}[(-1)(1)+(-2)(2)+(-3)(3)]=\frac{-14}{3}=-4.667$ Why this step? ${\theta }_0$'s gradient averages the raw residuals; ${\theta }_1$'s weights each residual by its $x_i$ (bigger $x$ pulls harder on the slope).
4. Update. ${\theta }_0:=0-0.1(-2)=0.2$, ${\theta }_1:=0-0.1(-4.667)=0.4667$. Why this step? We subtract the gradient, so a negative gradient pushes the parameter up — exactly toward the true $y=x$.

Verify: Cost before $=\frac{1}{6}(1+4+9)=2.333$. New predictions $h(x)=0.2+0.4667x$ give residuals $-0.333,-0.867,-1.4$; new cost $=\frac{1}{6}(0.111+0.751+1.96)=0.470$. Cost fell $2.333\to 0.470$ ✓ — healthy descent. Why this step? We recompute $J$ with the updated line to confirm the step actually went downhill; a lower cost is the only proof the update helped.

The figure below shows this single step: the grey flat line is the start ($y=0$), the butter vertical bars are the three residuals pulling the line up, the lavender line is where we land after one update, and the dashed mint line is the true $y=x$ we are heading toward. Notice the lavender line already tilts most of the way to mint — one healthy step covers a lot of ground.

---

Ex 2 — Cell B: learning rate too large (divergence)

Forecast: A giant step. Do we still land near $y=x$, or fly past it? Guess before reading.

1. Gradients are identical to Ex 1: $-2$ and $-4.667$ (same data, same start). Why this step? The gradient depends only on the current point on the surface, not on $\alpha$.
2. Update with $\alpha =1$. ${\theta }_0:=0-1(-2)=2$, ${\theta }_1:=0-1(-4.667)=4.667$. Why this step? One times a big gradient is a big leap — we hurl the slope from $0$ all the way to $4.667$.
3. New cost. Predictions $h(x)=2+4.667x$: $h(1)=6.667,h(2)=11.33,h(3)=16$. Residuals $5.667,9.33,13$. Cost $=\frac{1}{6}(32.1+87.1+169)=48.0$. Why this step? We overshot so hard the line is now steeper than needed and above every point.

Verify: Cost went $2.333\to 48.0$ — it increased 20-fold. This is divergence: with $\alpha$ too big we climbed the far wall of the valley. Fix by shrinking $\alpha$ (see 4.2.03-Learning-rate-schedules). Why this step? Recomputing $J$ after the step is how we detect divergence — a rising cost is the alarm bell that $\alpha$ is too large.

In the figure below, the grey line is again the start, but now the coral line (our one step with $\alpha =1$) shoots far above every data point — its slope $4.667$ badly overshoots the true slope $1$ (dashed mint). The annotation marks where it "flew over the valley": instead of settling into the bowl, we launched up the opposite wall, which is why the cost grew instead of shrinking.

---

Ex 3 — Cell C: learning rate too small

Forecast: Correct direction, but how far? Guess: ${\theta }_1$ barely moves.

1. Same gradients: $-2$, $-4.667$. Why this step? Data and start point are unchanged from Ex 1, and gradients ignore $\alpha$, so they are exactly the same numbers.
2. Update. ${\theta }_0:=0-0.001(-2)=0.002$, ${\theta }_1:=0-0.001(-4.667)=0.004667$. Why this step? Multiplying by a tiny $\alpha$ gives a tiny nudge — right direction, almost no progress.

Verify: New predictions $h(x)=0.002+0.004667x$ give $h(1)=0.006667,h(2)=0.01133,h(3)=0.016$; residuals $-0.9933,-1.9887,-2.984$; new cost $=\frac{1}{6}(0.9867+3.955+8.904)=2.308$. It dropped from $2.333$ but only by $\approx 1\%$ — at this rate we'd need hundreds of iterations. Why this step? We plug the updated parameters back into the same MSE formula to confirm the cost did move in the right direction; the tiny drop is the fingerprint of the classic "too small" symptom — correct but painfully slow.

---

Ex 4 — Cell D: data already perfectly on a line (zero residual)

Forecast: We're already sitting in the valley floor. What should the gradient be?

1. Predictions. $h(1)=1+2=3,h(2)=5,h(3)=7$ — exact. Why this step? Plugging the true parameters reproduces the data.
2. Residuals. $e_1=e_2=e_3=0$. Why this step? Perfect predictions leave zero gap.
3. Gradients. Both sums contain only $0$'s $\Rightarrow \partial J/\partial {\theta }_0=\partial J/\partial {\theta }_1=0$. Why this step? Every term is (residual)$\times$something, and residual $=0$.
4. Update. ${\theta }_j:={\theta }_j-\alpha \cdot 0={\theta }_j$ — nothing moves. Why this step? A zero gradient times any $\alpha$ is zero, so the parameters stay put — this is a fixed point.

Verify: Cost $J=\frac{1}{6}(0)=0$, the global minimum. This is the fixed point of gradient descent: zero gradient $\Rightarrow$ no update, regardless of $\alpha$. This is why a convex bowl (see 3.1.02-Convex-optimization) has exactly one resting place. Why this step? Confirming $J=0$ proves we are literally on the valley floor — there is nowhere lower to go, so descent correctly does nothing.

---

Ex 5 — Cell E: degenerate data, all $x$ equal

Forecast: Can we ever pin down a slope when the input never varies? Think first.

1. Predictions all $=0$; residuals $-1,-3,-5$. Why this step? With both parameters zero the line is flat, so each prediction is $0$ and the residual is just $-y_i$.
2. Gradients. $\partial J/\partial {\theta }_0=\frac{1}{3}(-1-3-5)=-3$. $\partial J/\partial {\theta }_1=\frac{1}{3}[(-1)(2)+(-3)(2)+(-5)(2)]=\frac{-18}{3}=-6$. Why this step? Both gradients are non-zero, so the parameters will move.
3. Update. ${\theta }_0:=0.3,{\theta }_1:=0.6$. Why this step? Subtract $\alpha$ times each gradient; both go up because both gradients are negative.

Verify: Both change, but note ${\theta }_1$ is meaningless here: the model only ever sees $x=2$, so $h={\theta }_0+2{\theta }_1$. Only the combination ${\theta }_0+2{\theta }_1$ is determined. Check: after the step ${\theta }_0+2{\theta }_1=0.3+1.2=1.5$, moving toward $\bar{y}=3$. Why this step? Tracking the combination ${\theta }_0+2{\theta }_1$ (not each parameter alone) is the only meaningful sanity check when the slope is unidentifiable — infinitely many $({\theta }_0,{\theta }_1)$ pairs give the same fit. Feature variation is required (this is exactly what 2.2.05-Normal-equation would show as a singular matrix, and why 2.06-Feature-scaling-and-normalization matters).

---

Ex 6 — Cell F: negative-slope data (sign check)

Forecast: Truth has a negative slope. Should ${\theta }_1$ go up or down on step one?

1. Predictions $0,0,0$; residuals $-5,-3,-1$. Why this step? All predictions are below the data, so all residuals negative.
2. Gradients. $\partial J/\partial {\theta }_0=\frac{1}{3}(-9)=-3$. $\partial J/\partial {\theta }_1=\frac{1}{3}[(-5)(1)+(-3)(2)+(-1)(3)]=\frac{-14}{3}=-4.667$. Why this step? Weighting the negative residuals by $x_i$ still gives a negative sum, so this early gradient does not yet "know" the slope should be negative.
3. Update. ${\theta }_0:=0.3,{\theta }_1:=0.4667$. Why this step? First step both go up — even though the true slope is negative! The gradient reacts to the current error, not the final answer.

Verify: After this step the line over-tilts up, but the next iterations will pull ${\theta }_1$ back down past zero toward $-2$. Predictions $0.767,1.233,1.700$; residuals $-4.233,-1.767,0.700$; new cost $=\frac{1}{6}(17.92+3.12+0.49)=3.59$, down from $\frac{1}{6}(25+9+1)=5.83$ ✓. Why this step? We plug back into the MSE formula to confirm the cost genuinely fell, proving the "wrong-looking" up-step is still downhill on the surface — direction on the cost surface, not the sign of the final slope, is what governs a single step.

In the figure below, the grey start line ($y=0$) sits below all three coral points. The lavender line (after one step) tilts upward — the annotation flags this: step 1 tilts up even though the true line (dashed mint $y=-2x+7$) slopes down. This is the whole lesson of Cell F: a single gradient step follows the current residuals, and only over many steps does ${\theta }_1$ swing down through zero to the correct negative slope.

---

Ex 7 — Cell G: single point, $n=1$ (degenerate limit)

Forecast: How many lines pass exactly through one point? (Answer: infinitely many.) So what does descent do?

1. Prediction $h(4)=0$; residual $e_1=-8$. Why this step? With flat start the prediction is $0$, so the gap to $y=8$ is $-8$.
2. Gradients ($n=1$, no averaging needed): $\partial J/\partial {\theta }_0=-8$, $\partial J/\partial {\theta }_1=(-8)(4)=-32$. Why this step? With one example the $\frac{1}{n}$ is $1$; the slope gradient is inflated by the large $x=4$.
3. Update. ${\theta }_0:=0.8,{\theta }_1:=3.2$. Why this step? Subtract $\alpha$ times each gradient; the big slope gradient produces a big jump in ${\theta }_1$.

Verify: New prediction $h(4)=0.8+3.2(4)=13.6$, residual $+5.6$ — we overshot $8$! The true minimum is any line with ${\theta }_0+4{\theta }_1=8$ (a whole line of solutions, cost $0$). Why this step? Plugging the updated line back through the single point exposes the overshoot ($-8\to +5.6$, the residual flipped sign), and it shows the fit is under-determined — with one point and this $\alpha$ the effective step is large because $x=4$ inflates the slope gradient. Lesson: $n=1$ has no unique fit; you need at least two distinct $x$ values.

---

Ex 8 — Cell H: real-world word problem

Forecast: Slope $\approx$ "extra coffees per barista" $\approx 20$. Guess ${\theta }_1$ heads toward $\sim 20$.

1. Predictions $0,0,0$; residuals $-20,-38,-61$ (coffees). Why this step? Units: residuals are in coffees, the same units as $y$ — every quantity that flows out of a residual (both gradients) therefore also carries a coffee-unit, which is exactly why $\alpha$ must be chosen small.
2. Unit ⟷ learning-rate reasoning. Because $y$ values are large ($20$–$61$), residuals are large, so the gradients are large ($\sim -40,-93$). A step is $\alpha \times \text{gradient}$; with a "normal" $\alpha =0.1$ the slope would jump by $\sim 9$ in one step and overshoot wildly. Why this step? The size of the gradient scales with the units and magnitude of your data. Big-valued targets demand a smaller $\alpha$ (here $0.01$) or, better, feature/target scaling so a single $\alpha$ works regardless of units.
3. Gradients. $\partial J/\partial {\theta }_0=\frac{1}{3}(-20-38-61)=\frac{-119}{3}=-39.67$. $\partial J/\partial {\theta }_1=\frac{1}{3}[(-20)(1)+(-38)(2)+(-61)(3)]=\frac{-279}{3}=-93$. Why this step? ${\theta }_0$'s gradient averages the raw coffee residuals; ${\theta }_1$'s weights each residual by the barista count.
4. Update. ${\theta }_0:=0-0.01(-39.67)=0.3967$, ${\theta }_1:=0-0.01(-93)=0.93$. Why this step? We subtract $\alpha$ times each gradient; the small $\alpha =0.01$ keeps the slope jump modest ($0\to 0.93$) so we creep stably toward the true $\sim 20$ instead of overshooting.
5. Post-update predictions (sanity peek). $h(1)=0.3967+0.93=1.33$, $h(2)=2.26$, $h(3)=3.19$ — still far below the data, as expected after just one gentle step. Why this step? Checking predictions confirms we moved in the right direction (all predictions rose from $0$) without any wild overshoot — the small-$\alpha$ choice paid off.

Verify (fitted answer): Running to convergence, the least-squares line has slope ${\theta }_1=20.5$ and intercept ${\theta }_0=-1.333$. Derivation: $\bar{x}=2,\bar{y}=39.667$; ${\theta }_1=\frac{\sum (x-\bar{x})(y-\bar{y})}{\sum (x-\bar{x}{)}^2}=\frac{41}{2}=20.5$, then ${\theta }_0=\bar{y}-{\theta }_1\bar{x}=39.667-20.5(2)=-1.333$. Prediction for 4 baristas: $h(4)=-1.333+20.5(4)=80.67\approx 81$ coffees. Why this step? We solve for the exact optimum (the same line 2.2.05-Normal-equation gives) to state the forecast the problem asked for; the units check — coffees/barista $\times$ baristas $=$ coffees ✓ — confirms the prediction is dimensionally sensible, and $80.67$ sits just above the last data point $61$, matching a rising trend.

---

Ex 9 — Cell I: exam twist — start already at the optimum

Forecast: At the minimum the gradient should be zero — so even a huge $\alpha$ does nothing. True or false?

1. Predictions. $h(1)=19.167,h(2)=39.667,h(3)=60.167$. Why this step? Plug the fitted line into each $x$ to get the best-fit predictions.
2. Residuals. $e_1=-0.833,e_2=1.667,e_3=-0.833$. Why this step? Least squares makes residuals sum to zero and their $x$-weighted sum zero — that's the defining condition.
3. Gradients. $\partial J/\partial {\theta }_0=\frac{1}{3}(-0.833+1.667-0.833)\approx 0$. $\partial J/\partial {\theta }_1=\frac{1}{3}[(-0.833)(1)+(1.667)(2)+(-0.833)(3)]=0$. Why this step? Both defining conditions of least squares force these sums to zero.
4. Update. ${\theta }_j:={\theta }_j-0.5\cdot 0={\theta }_j$ — no change. Why this step? Anything times a zero gradient is zero, so $\alpha$ size is irrelevant at the minimum.

Verify: Even with $\alpha =0.5$ (large), nothing moves because both gradients are $0$. This confirms the least-squares point is the fixed point of gradient descent — the same solution 2.2.05-Normal-equation gives in one shot. Residuals do not vanish here (data isn't perfectly linear), but their weighted sums do, which is what "minimum" means. Why this step? Verifying the gradients are zero (not the residuals) is the exam trap: a big $\alpha$ is harmless only because the gradient — not the residual — is what gets multiplied by $\alpha$, and at the optimum that gradient is exactly zero.

---

Recall Self-check

One step of GD on divergent $\alpha$ makes cost go which way? ::: Up — it increases (Ex 2 went $2.33\to 48$). Why does gradient descent stop at the least-squares line even with big $\alpha$? ::: The gradient is exactly zero there, so every update subtracts $\alpha \cdot 0=0$. With all $x_i$ equal, which parameter is unidentifiable? ::: The slope ${\theta }_1$ — only the combination ${\theta }_0+\bar{x}{\theta }_1$ is determined. First GD step on negative-slope data pushes ${\theta }_1$ which way from $0$? ::: Up (positive), because early residuals are all negative; it corrects later.

See also: 2.2.03-Hypothesis-representation-in-linear-regression, 2.3.02-Logistic-cost-function, 2.4.01-Gradient-descent-for-neural-networks, 4.2.01-Stochastic-gradient-descent.