Polynomial regression: Worked examples — AI-ML deep dive

This is a worked-examples workshop for Polynomial regression. The parent note built the theory: transform a feature $x$ into powers $1, x, x^{2}, \dots, x^{d}$ , then run ordinary linear regression on those new columns. Here we hit every kind of situation that can appear — small quadratics, degenerate data, negative inputs, scaling disasters, over-fitting, real-world word problems, and exam twists — so no scenario ever surprises you.

Four symbols you must have straight before we start (the parent defined them, we re-anchor them here):

Definition The objects we keep re-using

$d$ (read "dee") — the degree of the polynomial: the highest power of $x$ we include. Degree 1 is a straight line, degree 2 a parabola, degree 3 an S-curve, and so on. A degree- $d$ model has $d + 1$ parameters (one for each of $1, x, x^{2}, \dots, x^{d}$ ).
$Φ$ (read "phi", capital) — the design matrix. One row per data point, one column per feature. For a degree- $d$ fit of a single input, row $i$ is $[1, x^{(i)}, (x^{(i)})^{2}, \dots, (x^{(i)})^{d}]$ , so $Φ$ has $m$ rows and $d + 1$ columns.
$θ$ (read "theta") — the parameter vector, one number per column of $Φ$ . It is what we solve for.
$y$ (read "why", bold) — the target vector, the list of observed outputs: $y = [y^{(1)}, y^{(2)}, \dots, y^{(m)}]^{T}$ , one entry per data point. This is the "right answer" we want $h_{θ}$ to reproduce.
$m$ — the number of data points (rows of $Φ$ , entries of $y$ ). We will refer to it often.

The whole game is the normal equation $θ = (Φ^{T} Φ)^{- 1} Φ^{T} y$ : "combine all the data ( $Φ^{T} Φ$ , $Φ^{T} y$ ) and undo the combination ( $(\cdot)^{- 1}$ ) to recover the best $θ$ ."

The scenario matrix

Every cell below is covered by at least one worked example. If a situation is not in this table, it is a combination of these — you will be able to build it.

#	Case class	What makes it tricky	Example
A	Exact quadratic fit, positive $x$	3 params, 3+ clean points	Ex 1
B	Noisy data, best-fit not exact	fit passes near, not through	Ex 2
C	Negative + zero inputs	signs, $x = 0$ kills higher powers	Ex 3
D	Degenerate / singular $Φ^{T} Φ$	duplicate $x$ or too few points	Ex 4
E	Scaling catastrophe	$x$ vs $x^{3}$ off by orders of magnitude	Ex 5
F	Over-fit limit ( $d + 1 = m$ )	zero train error, wild between points	Ex 6
G	Real-world word problem	choose degree, interpret coefficients	Ex 7
H	Exam twist: interaction term	two features, cross term $x_{1} x_{2}$	Ex 8

A — Exact quadratic, clean positive data

Worked example Example 1 — three points, one parabola

Data: $(1, 2), (2, 5), (3, 10)$ . Fit $h (x) = θ_{0} + θ_{1} x + θ_{2} x^{2}$ (degree $d = 2$ ).

Forecast: three unknowns, three points → guess we can hit all three exactly. What is the parabola?

Build $Φ$ and $y$ . Why this step? Each row is $[1, x, x^{2}]$ — the recipe for degree 2. $Φ = 111123149, y = 2510$
Here $Φ$ is square (3×3) and invertible, so instead of the full normal equation we can solve $Φ θ = y$ directly. Why? When rows = columns and points are distinct, there is a unique exact solution; least-squares reduces to plain solving.
Solve by elimination. Write the three equations: $θ_{0} + θ_{1} + θ_{2} = 2, θ_{0} + 2 θ_{1} + 4 θ_{2} = 5, θ_{0} + 3 θ_{1} + 9 θ_{2} = 10.$ Subtract eqn 1 from eqn 2: $θ_{1} + 3 θ_{2} = 3$ . Subtract eqn 2 from eqn 3: $θ_{1} + 5 θ_{2} = 5$ . Subtract those two: $2 θ_{2} = 2 \Rightarrow θ_{2} = 1$ . Back-substitute: $θ_{1} + 3 = 3 \Rightarrow θ_{1} = 0$ ; then $θ_{0} + 0 + 1 = 2 \Rightarrow θ_{0} = 1$ . Why elimination? Cancelling the shared intercept column first isolates the higher-order terms one at a time. $h (x) = 1 + 0 \cdot x + 1 \cdot x^{2} = x^{2} + 1$ Why this shape? $2 = 1^{2} + 1$ , $5 = 2^{2} + 1$ , $10 = 3^{2} + 1$ — the data was $x^{2} + 1$ .

Verify: $h (1) = 2$ ✓, $h (2) = 5$ ✓, $h (3) = 10$ ✓. Zero training error, as forecast.

B — Noisy data, best-fit only approximate

Worked example Example 2 — four points, three parameters, no exact fit

Data: $(0, 1), (1, 2), (2, 9), (3, 10)$ . Fit a quadratic ( $d = 2$ ).

Forecast: 4 points, 3 params → cannot pass through all four. Least squares finds the parabola with smallest total squared miss.

Build $Φ$ . Why? Rows $[1, x, x^{2}]$ again; note first row uses $x = 0$ so it is $[1, 0, 0]$ . $Φ = 111101230149, y = 12910$
Form $Φ^{T} Φ$ and $Φ^{T} y$ . Why? The normal equation compresses all four rows into these small sums. Each entry of $Φ^{T} y$ is a column-of- $Φ$ dotted with $y$ : first entry $\sum y = 1 + 2 + 9 + 10 = 22$ ; second $\sum x y = 0 + 2 + 18 + 30 = 50$ ; third $\sum x^{2} y = 0 + 2 + 36 + 90 = 128$ . $Φ^{T} Φ = 461461436143698, Φ^{T} y = 2250128$
Solve the $3 \times 3$ system $Φ^{T} Φ θ = Φ^{T} y$ . Why by elimination, not literal inversion? Because inverting a matrix by hand is more work and less numerically stable than solving the linear system directly (this is exactly what a computer's solve does — it never actually forms $(Φ^{T} Φ)^{- 1}$ ). We only write $(Φ^{T} Φ)^{- 1}$ to name the answer; we compute it by row-reduction. Doing so here yields $θ = [\frac{9}{5}, - \frac{7}{5}, \frac{5}{2}]^{T} = [1.8, - 1.4, 2.5]^{T} .$ Why these fractions? They are the exact rational solution of the three equations $4 θ_{0} + 6 θ_{1} + 14 θ_{2} = 22$ , $6 θ_{0} + 14 θ_{1} + 36 θ_{2} = 50$ , $14 θ_{0} + 36 θ_{1} + 98 θ_{2} = 128$ .
Why not exact? With more equations than unknowns, the system is over-determined; the fit balances errors instead of nailing every point.

Verify: predictions $h (0) = 1.8, h (1) = 1.8 - 1.4 + 2.5 = 2.9, h (2) = 1.8 - 2.8 + 10.0 = 9.0, h (3) = 1.8 - 4.2 + 22.5 = 20.1$ . Residuals ( $1 - 1.8, 2 - 2.9, 9 - 9.0, 10 - 20.1$ ) are nonzero and balance around zero — exactly the "approximate" behaviour the parent note promised for real data.

C — Negative and zero inputs

Worked example Example 3 — signs matter

Data: $(- 2, 5), (- 1, 2), (0, 1), (1, 2), (2, 5)$ . Fit a quadratic ( $d = 2$ ).

Forecast: symmetric U-shape around $x = 0$ . A symmetric parabola has no linear term ( $θ_{1} = 0$ ). Guess $θ_{1} = 0, θ_{2} = 1, θ_{0} = 1$ .

Build $Φ$ , being careful with negatives: $(- 2)^{2} = + 4$ , $(- 1)^{2} = + 1$ . Why care? Squaring erases sign — that is exactly why the parabola is left–right symmetric. $Φ = 11111 - 2 - 1 012 41014, y = 52125$
Form the normal-equation pieces. Why? We have 5 points and 3 params, so this is a genuine least-squares problem — we must go through $Φ^{T} Φ$ and $Φ^{T} y$ . Column sums: $\sum 1 = 5$ , $\sum x = 0$ (symmetric!), $\sum x^{2} = 10$ , $\sum x^{3} = 0$ (odd powers cancel), $\sum x^{4} = 34$ . And $\sum y = 15$ , $\sum x y = 0$ , $\sum x^{2} y = 54$ . So $Φ^{T} Φ = 5010010010034, Φ^{T} y = 15054 .$
Solve. Why does it split so cleanly? Because the middle row/column is decoupled: row 2 reads $10 θ_{1} = 0 \Rightarrow θ_{1} = 0$ immediately (the symmetry killed the odd term). The remaining $2 \times 2$ system $5 θ_{0} + 10 θ_{2} = 15$ , $10 θ_{0} + 34 θ_{2} = 54$ solves to $θ_{2} = 1$ , $θ_{0} = 1$ . So $θ = [1, 0, 1]^{T}$ and $h (x) = x^{2} + 1$ . Why exact here? The five points happen to lie perfectly on $x^{2} + 1$ , so the residual is zero even though it is a least-squares fit.
The $x = 0$ point contributes row $[1, 0, 0]$ : only the intercept survives. Why? At $x = 0$ every positive power vanishes, so $h (0) = θ_{0}$ . That is how the intercept is "read off" the data.

Verify: $h (- 2) = 5$ ✓, $h (- 1) = 2$ ✓, $h (0) = 1$ ✓, $h (1) = 2$ ✓, $h (2) = 5$ ✓.

D — Degenerate input: when $Φ^{T} Φ$ can't be inverted

Worked example Example 4 — duplicate

$x$ , singular matrix Data: $(1, 3), (1, 5), (2, 4)$ . Try to fit a quadratic ( $θ_{0}, θ_{1}, θ_{2}$ ).

Forecast: we have 3 points but only 2 distinct $x$ -values. A degree-2 curve needs 3 distinct $x$ 's to be pinned down. Guess: the solve fails.

Build $Φ$ . Why look closely? Rows 1 and 2 are identical: $[1, 1, 1]$ and $[1, 1, 1]$ . $Φ = 111112114$
Form $Φ^{T} Φ$ so we can test invertibility. Column sums give $Φ^{T} Φ = 346461061018 .$
Compute $det (Φ^{T} Φ)$ by cofactor expansion along row 1. Why the determinant? A square matrix is invertible iff its determinant is nonzero — a zero determinant is the algebraic fingerprint of "columns are linearly dependent." $det = 3 (6 \cdot 18 - 10 \cdot 10) - 4 (4 \cdot 18 - 10 \cdot 6) + 6 (4 \cdot 10 - 6 \cdot 6)$ $= 3 (108 - 100) - 4 (72 - 60) + 6 (40 - 36) = 3 (8) - 4 (12) + 6 (4) = 24 - 48 + 24 = 0.$ Why does it vanish? Column 3 of $Φ^{T} Φ$ is a linear combination of columns 1 and 2 — inherited from the fact that in $Φ$ the $x^{2}$ column carries no information beyond the $x$ column when only two distinct $x$ 's appear. Duplicate/collinear inputs $\Rightarrow$ dependent columns $\Rightarrow$ zero determinant.
The fix (as in regularization): add a tiny ridge term $(Φ^{T} Φ + λ I)^{- 1}$ (which nudges the determinant off zero), or lower the degree so params $\leq$ distinct points.

Verify: $det (Φ^{T} Φ) = 0$ — the machine check confirms singularity. Lesson: degree must respect how many distinct inputs you have.

E — The scaling catastrophe

Worked example Example 5 — same fit, two conditioning worlds

Take $x$ -values ${10, 20, 30}$ and a degree-3 model ( $d = 3$ ). Compare raw features vs scaled features.

Forecast: raw powers span $x \in [10, 30]$ but $x^{3} \in [1000, 27000]$ — a 27× internal spread. Guess: $Φ^{T} Φ$ is badly ill-conditioned before scaling.

Raw $Φ$ columns: $[1], [x], [x^{2}], [x^{3}]$ have magnitudes $1, \sim 20, \sim 400, \sim 8000$ .

Read the figure: each bar is the typical size of one feature column, plotted on a log scale (so equal bar-heights would mean equal magnitudes). Three bars are black; the red $x^{3}$ bar towers over the "1" column by a factor of tens of thousands. That single dominating column is the whole problem: matrix inversion and gradient descent both treat a giant column as "more important" purely because of its size, not its usefulness. 2. Measure conditioning properly with the condition number, not the determinant. Why not the determinant? The determinant just multiplies all the scales together, so it explodes or vanishes with units and tells you little about numerical stability. The right tool is the condition number $κ (Φ^{T} Φ) = λ_{m a x} / λ_{m i n}$ (ratio of largest to smallest eigenvalue): it measures how much a tiny change in $y$ can blow up in $θ$ . A large $κ$ means "near-singular in practice." For the raw features here $κ$ is astronomically large (far above $1 0^{8}$ ). 3. Scale each column by $x_{j} \leftarrow (x_{j} - μ_{j}) / σ_{j}$ . Why? This forces every feature to mean 0, spread 1, so no column dominates and every bar in the figure would shrink to the same height. 4. Recompute $κ$ on scaled features: it drops to a small number ( $O (1)$ – $O (10)$ ) — the problem becomes well-behaved and both the normal equation and gradient descent behave.

Verify: the machine check confirms $κ$ of the raw $Φ^{T} Φ$ exceeds $1 0^{8}$ while the scaled version is below $100$ — exactly the parent's "always scale after transforming."

F — The over-fit limit

Worked example Example 6 — degree 4 through 5 points

Data: $(0, 0), (1, 1), (2, 0), (3, 1), (4, 0)$ — a zig-zag ( $m = 5$ points). Fit degree $d = 4$ .

Forecast: 5 params ( $θ_{0} \dots θ_{4}$ ) for 5 points → the model has exactly enough freedom to hit every point ( $d + 1 = m$ ). Guess: zero training error, but a wildly wiggling curve.

$Φ$ is 5×5 and (for distinct $x$ 's) invertible. Why? Distinct inputs give a Vandermonde matrix with nonzero determinant.
Solve $Φ θ = y$ directly. The unique interpolating quartic passes through all five points.

Read the figure: the black dots are the five data points; the red curve is the degree-4 fit. Notice it does pass exactly through every dot (zero training error) — but look between the dots: the red curve swings far above and below, overshooting where there is no data. That overshoot is variance: the model spent its freedom memorising the sample rather than learning a smooth trend. 3. Why it swings: with no residual freedom left, the polynomial has nothing to "spend" on smoothness, so it oscillates. This is the high-variance end of the bias–variance tradeoff. 4. The remedy is a validation split: pick the degree with lowest validation error, not lowest train error.

Verify: the fitted quartic reproduces all five $y$ -values exactly — training error $= 0$ , confirming the over-fit.

G — Real-world word problem

Worked example Example 7 — stopping distance of a car

A car's braking distance $y$ (metres) versus speed $x$ (m/s): $(10, 15), (20, 45), (30, 95)$ . Physics says stopping distance $\approx$ reaction distance (linear in $x$ ) $+$ braking distance ( $\propto x^{2}$ ). So model $h (x) = θ_{0} + θ_{1} x + θ_{2} x^{2}$ (degree $d = 2$ ).

Forecast: the $x^{2}$ term should dominate at high speed; $θ_{2} > 0$ . Three points, three params → exact fit.

Why a quadratic and not a cubic? Why this step? Domain knowledge: kinetic energy $\propto x^{2}$ , and energy sets braking distance. We pick degree from physics, avoiding the Ex 6 over-fit trap.
Build the 3×3 system. $Φ = 111102030100400900, y = 154595$
Solve by elimination. The three equations are $θ_{0} + 10 θ_{1} + 100 θ_{2} = 15$ , $θ_{0} + 20 θ_{1} + 400 θ_{2} = 45$ , $θ_{0} + 30 θ_{1} + 900 θ_{2} = 95$ . Subtract eqn 1 from eqn 2: $10 θ_{1} + 300 θ_{2} = 30$ . Subtract eqn 2 from eqn 3: $10 θ_{1} + 500 θ_{2} = 50$ . Subtract those: $200 θ_{2} = 20 \Rightarrow θ_{2} = 0.1$ . Then $10 θ_{1} + 30 = 30 \Rightarrow θ_{1} = 0$ ; and $θ_{0} + 0 + 10 = 15 \Rightarrow θ_{0} = 5$ . $h (x) = 5 + 0 \cdot x + 0.1 x^{2}$
Interpret coefficients (units check): $θ_{0} = 5$ m is a baseline offset; there is no linear reaction term ( $θ_{1} = 0$ ) in this clean dataset; $0.1 x^{2}$ is the pure braking term, growing with the square of speed. Why interpret? A model you can read is a model you can trust.

Verify: $h (10) = 5 + 10 = 15$ ✓, $h (20) = 5 + 40 = 45$ ✓, $h (30) = 5 + 90 = 95$ ✓. Exact fit, as forecast, and $θ_{2} > 0$ .

H — Exam twist: interaction term

Worked example Example 8 — two features, degree 2 with cross term

Predict crop yield $y$ from rainfall $x_{1}$ and heat $x_{2}$ : model $h = θ_{0} + θ_{1} x_{1} + θ_{2} x_{2} + θ_{3} x_{1} x_{2} .$ Data: $(x_{1}, x_{2}, y) = (1, 1, 5), (2, 1, 7), (1, 2, 8), (2, 2, 12)$ .

Forecast: the $x_{1} x_{2}$ term captures "rain AND heat together". Four points, four params → exact.

Build $Φ$ with an interaction column $x_{1} x_{2}$ . Why? Pure powers alone can't model synergy between the two inputs. $Φ = 1111121211221224, y = 57812$
Solve by elimination. From the four equations, subtract eqn 1 from eqn 2 (only $x_{1}$ and $x_{1} x_{2}$ change): $θ_{1} + θ_{3} = 2$ . Subtract eqn 1 from eqn 3 (only $x_{2}$ and $x_{1} x_{2}$ change): $θ_{2} + θ_{3} = 3$ . Subtract eqn 2 from eqn 4: $θ_{2} + 2 θ_{3} = 5$ . Subtract that last pair: $θ_{3} = 2$ . Back-substitute: $θ_{1} = 0$ , $θ_{2} = 1$ , and eqn 1 gives $θ_{0} + 0 + 1 + 2 = 5 \Rightarrow θ_{0} = 2$ . $h = 2 + 0 \cdot x_{1} + 1 \cdot x_{2} + 2 x_{1} x_{2}$
Read the cross term: $θ_{3} = 2 > 0$ means rain and heat reinforce yield — the synergy is real and strong.

Verify: $h (1, 1) = 2 + 0 + 1 + 2 = 5$ ✓, $h (2, 1) = 2 + 0 + 1 + 4 = 7$ ✓, $h (1, 2) = 2 + 0 + 2 + 4 = 8$ ✓, $h (2, 2) = 2 + 0 + 2 + 8 = 12$ ✓.

Recall Quick self-test

Which case class has a singular $Φ^{T} Φ$ ? ::: Case D — duplicate/too-few distinct $x$ -values. Why must you scale before a degree-3 fit? ::: Raw $x^{3}$ dwarfs $x$ , blowing up the condition number of $Φ^{T} Φ$ (Case E). When does degree $d$ give exactly zero training error? ::: When $d + 1 \geq m$ , where $m$ is the number of data points (Case F). What forces $θ_{1} = 0$ in Example 3? ::: The data is symmetric about $x = 0$ , so the odd (linear) term cancels.

Read the Hinglish companion: 2.2.06 Polynomial regression (Hinglish).