Polynomial regression: Exercises — AI-ML deep dive

Intuition How to Use This Page

These are graded practice problems, from "just recognise the idea" up to "put everything together." Each problem has a full worked solution hidden inside a collapsible callout — try the problem first, then reveal. After each level there is a warning about the exact trap students fall into at that level. Everything here builds on the parent Polynomial regression note.

Before we begin, one reminder of the notation we will reuse everywhere:

$x$ is the input (one number, e.g. temperature).
$y$ is the target (the number we want to predict).
$m$ is the number of training examples (how many $(x, y)$ pairs we have).
$h_{θ} (x)$ is our hypothesis — the prediction the model makes for input $x$ .
$θ_{0}, θ_{1}, θ_{2}, \dots$ are the parameters (also called weights). These are the knobs we turn to fit the data.
The superscript $(i)$ as in $x^{(i)}$ means "the $i$ -th training example," not a power. Powers are written $x^{2}, x^{3}$ .
$Φ$ (capital Greek "phi") is the design matrix: one row per example, one column per feature.

The single picture below is our running mental model for the whole page: curved data that a straight line cannot follow but a polynomial can. Every exercise is a variation on this theme — building the features, fitting them, and choosing how much curve is too much.

Notice the coral dots bending upward: a plain lavender line leaves big gaps, while the mint quadratic follows the bend. Keep this contrast in mind as you work through the levels.

Level 1 — Recognition

Recall Solution L1.1

Answer: (b).

(a) is degree 1 (a straight line) — highest power of $x$ is $1$ .
(b) has powers up to $x^{3}$ → degree 3. ✓
(c) uses $sin$ , not powers of $x$ , so it is not a polynomial.
(d) has the parameter $θ_{1}$ sitting in the exponent. That makes it non-linear in the parameters, so it is not linear/polynomial regression at all.

Key idea: "polynomial regression" = the model is a sum of powers of $x$ , each multiplied by a parameter. The powers can be as high as we like, but the parameters must appear only as plain multipliers.

Recall Solution L1.2

False. It uses the same algorithm. The word "linear" in linear regression means linear in the parameters $θ$ , not linear in $x$ . We simply create new features $x_{1} = x, x_{2} = x^{2}, x_{3} = x^{3}$ and hand them to ordinary linear regression. The algorithm never knows those columns were built from powers.

Recall Solution L1.3

$d + 1$ parameters: $θ_{0}, θ_{1}, \dots, θ_{d}$ . The " $+ 1$ " is the intercept $θ_{0}$ (the constant term, which multiplies $x^{0} = 1$ ).

Level 2 — Application

Recall Solution L2.1

Each row is $[1, x^{(i)}, (x^{(i)})^{2}]$ . $Φ = 111123149, y = 236$ Row 1: $[1, 1, 1^{2}]$ . Row 2: $[1, 2, 2^{2}] = [1, 2, 4]$ . Row 3: $[1, 3, 3^{2}] = [1, 3, 9]$ . Three examples → three rows; degree 2 plus intercept → three columns.

Recall Solution L2.2

Substitute directly:

$h_{θ} (0) = 2 - 3 (0) + 0^{2} = 2$ .
$h_{θ} (1) = 2 - 3 (1) + 1^{2} = 2 - 3 + 1 = 0$ .
$h_{θ} (4) = 2 - 3 (4) + 4^{2} = 2 - 12 + 16 = 6$ .

Recall Solution L2.3

$x \in [1, 100]$
$x^{2} \in [1, 10000]$
$x^{3} \in [1, 1000000]$

The three columns differ by up to four orders of magnitude. In this stretched-out cost landscape, gradient descent is forced to use one step size for all directions: it takes tiny, painfully slow steps in the small-scale direction (the $x$ column, low curvature) while overshooting and zig-zagging in the large-scale direction (the $x^{3}$ column, high curvature). The result is slow, unstable convergence, and $Φ^{T} Φ$ becomes ill-conditioned for the normal equation. Scaling with $x_{j} \leftarrow \frac{x _{j} - μ _{j}}{σ _{j}}$ puts every column on mean $0$ , standard deviation $1$ , so all directions have comparable curvature and one step size works well for all of them.

Recall Solution L2.4

Mean: $μ = \frac{2 + 4 + 6 + 8}{4} = 5$ . Deviations: $- 3, - 1, 1, 3$ . Squared: $9, 1, 1, 9$ ; sum $= 20$ ; divide by $m = 4$ → variance $= 5$ ; $σ = 5 \approx 2.236$ . Standardised values:

$\frac{2 - 5}{5} = \frac{- 3}{2.236} \approx - 1.342$
$\frac{4 - 5}{5} \approx - 0.447$
$\frac{6 - 5}{5} \approx 0.447$
$\frac{8 - 5}{5} \approx 1.342$

Level 3 — Analysis

Recall Solution L3.1

Write $h_{θ} (x) = θ_{0} + θ_{1} x + θ_{2} x^{2}$ and plug in each point: $θ_{0} + θ_{1} + θ_{2} θ_{0} + 2 θ_{1} + 4 θ_{2} θ_{0} + 3 θ_{1} + 9 θ_{2} = 2 (x = 1) = 3 (x = 2) = 6 (x = 3)$ Subtract eq1 from eq2: $θ_{1} + 3 θ_{2} = 1$ . Subtract eq2 from eq3: $θ_{1} + 5 θ_{2} = 3$ . Subtract those two: $2 θ_{2} = 2 \Rightarrow θ_{2} = 1$ . Back-substitute: $θ_{1} + 3 (1) = 1 \Rightarrow θ_{1} = - 2$ . Then $θ_{0} - 2 + 1 = 2 \Rightarrow θ_{0} = 3$ .

Result: $h_{θ} (x) = 3 - 2 x + x^{2}$ . Check: $h (1) = 3 - 2 + 1 = 2$ ✓, $h (2) = 3 - 4 + 4 = 3$ ✓, $h (3) = 3 - 6 + 9 = 6$ ✓. With 3 parameters and 3 points, the fit passes through all of them exactly.

Recall Solution L3.2

$d = 1$ → (i) — the straight line is too rigid to bend to the pattern: underfitting (high bias).
$d = 2$ → (ii) — enough flexibility to catch the curve without chasing noise: the sweet spot.
$d = 4$ → (iii) — with $d + 1 = 5$ parameters and $m = 5$ points, the curve threads exactly through every point (including its noise): overfitting (high variance). See 2.3.02-Overfitting-and-regularization.

The number of free parameters ( $d + 1$ ) reaching the number of data points ( $m$ ) is exactly when a model can memorise the data.

Recall Solution L3.3

Lavender straight line — underfit. It ignores the visible curvature of the points; large gaps between line and dots mean high training error.
Mint smooth curve — good fit. It follows the overall bend of the points and passes near (not through) each one; it will generalise to new inputs.
Coral wiggly curve — overfit. It contorts to hit every single point exactly, including the noisy ones, and swings wildly between them — new points landing in those swings would be badly predicted.

The tell-tale sign: a fit that touches every point perfectly but oscillates violently between them is memorising, not learning.

Level 4 — Synthesis

Recall Solution L4.1

Design matrix and targets: $Φ = 1111123414916, y = 371321$ $Φ^{T} Φ = 41030103010030100354, Φ^{T} y = 44140484$ Solving $Φ^{T} Φ θ = Φ^{T} y$ gives $θ = 111 \Rightarrow h_{θ} (x) = 1 + x + x^{2} .$ Verify: $h (1) = 3, h (2) = 7, h (3) = 13, h (4) = 21$ — all exact. (Even with 4 points and 3 parameters, the data happened to lie perfectly on $x^{2} + x + 1$ , so least-squares error is zero.)

Recall Solution L4.2

$h_{θ} (x_{1}, x_{2}) = θ_{0} + θ_{1} x_{1} + θ_{2} x_{2} + θ_{3} x_{1}^{2} + θ_{4} x_{2}^{2} + θ_{5} x_{1} x_{2}$ Feature columns: ${1, x_{1}, x_{2}, x_{1}^{2}, x_{2}^{2}, x_{1} x_{2}}$ → 6 parameters ( $θ_{0}$ through $θ_{5}$ ). The term $x_{1} x_{2}$ is the interaction: it lets the model express "the effect of $x_{1}$ depends on the level of $x_{2}$ " (e.g. rainfall and temperature together affecting crop yield).

Recall Solution L4.3

With degree $9$ there are $d + 1 = 10$ feature columns but only $m = 8$ rows. $Φ$ has more columns than rows, so its columns cannot all be independent — $Φ^{T} Φ$ (a $10 \times 10$ matrix) has rank at most $8$ and is therefore singular (no inverse). Even below that limit, un-scaled high powers make it near-singular. Two fixes:

Reduce the degree so that $d + 1 \leq m$ (fewer parameters than points), removing the rank deficiency.
Add regularization (ridge): solve $θ = (Φ^{T} Φ + λ I)^{- 1} Φ^{T} y$ . Here $λ$ (Greek "lambda") is a small positive number we choose, called the regularization strength, and $I$ is the identity matrix — a matrix with $1$ 's down the diagonal and $0$ 's elsewhere, the matrix version of the number $1$ . Adding $λ I$ nudges every diagonal entry of $Φ^{T} Φ$ upward, which makes the matrix invertible again and curbs overfitting — see 2.3.02-Overfitting-and-regularization. (Also scale features first, per 2.2.05-Feature-scaling.)

Level 5 — Mastery

Recall Solution L5.1

Split the data into train / validation / test (2.4.01-Train-validation-test-split). The test set is locked away until the very end.
For each degree $d$ : build polynomial features on the training set, scale each column (2.2.05-Feature-scaling) using the training mean/std (then apply the same $μ, σ$ to validation/test — never re-fit them).
Train the model (normal equation or gradient descent) on the training set.
Record $J_{train}$ and $J_{val}$ (the cost on the training and validation sets).
Plot both curves against degree. Training error falls monotonically; validation error typically dips then rises (a U-shape) — the classic bias-variance signature.
Choose the degree at the minimum of the validation curve — that is the sweet spot where the model is complex enough to catch the pattern but not so complex that it starts chasing noise.
Report the final, honest performance by evaluating the chosen model once on the untouched test set. Because the test set played no part in choosing the degree, this number is an unbiased estimate of real-world performance.

This uses validation — not training error — as the judge, exactly because low training error can be pure memorisation.

Worked example Exercise L5.2

Given the (illustrative) error table below, pick the best degree and justify it using bias-variance language.

degree $d$	$J_{train}$	$J_{val}$
1	0.80	0.85
2	0.30	0.34
3	0.12	0.15
4	0.05	0.28
5	0.01	0.60

Recall Solution L5.2

Choose $d = 3$ . It has the lowest validation error (0.15).

$d = 1, 2$ : both errors high → the model is too simple, high bias / underfitting.
$d = 3$ : training and validation errors are both low and close together → good balance, generalises.
$d = 4, 5$ : training error keeps dropping (0.05, 0.01) but validation error rises sharply (0.28, 0.60) → the model is fitting noise, high variance / overfitting.

The widening gap between $J_{train}$ and $J_{val}$ past degree 3 is the tell-tale overfitting signal. The figure below sketches this U-shape.

Recall Solution L5.3

Polynomial regression applies a fixed feature map $ϕ (x) = [1, x, x^{2}, \dots, x^{d}]$ that turns one input into several. The prediction is $h_{θ} (x) = θ^{T} ϕ (x)$ — a plain weighted sum of those transformed features. Because the parameters $θ$ enter only as multipliers (never inside a power, sine, etc.), the cost $J (θ)$ is a quadratic bowl in $θ$ with a single minimum, and setting $\nabla_{θ} J = 0$ gives the same normal equation $θ = (Φ^{T} Φ)^{- 1} Φ^{T} y$ . The curvature lives entirely in the fixed feature columns of $Φ$ ; the learning is still a linear least-squares problem. Change $ϕ$ and you change the shape of the fit, without ever changing the algorithm.

Recall Self-Test (reveal after attempting)

Polynomial regression is linear in what? ::: In the parameters $θ$ — not in $x$ . A degree- $d$ single-feature model has how many parameters? ::: $d + 1$ . What single quantity should decide the degree? ::: The validation error (lowest wins). Why scale after making powers, not before? ::: So each engineered column ( $x, x^{2}, x^{3}$ ) is standardised on its own scale, keeping $Φ^{T} Φ$ well-conditioned.