Visual walkthrough — Polynomial regression

---

Step 1 — The problem a straight line cannot solve

WHAT. We have a scatter of points. Each point is a pair: a horizontal position we call $x$ (the input) and a vertical position we call $y$ (the output we want to predict). A straight line is any rule of the form $y={\theta }_0+{\theta }_{1}x$, where ${\theta }_0$ is the height where the line crosses the vertical axis and ${\theta }_1$ is its slope (how much $y$ climbs for each step right in $x$).

WHY. Before inventing a fancy tool, we must see why the simple one fails. Look at the figure: the dots bend upward, curving away from any straight line you could draw. No choice of ${\theta }_0$ (height) or ${\theta }_1$ (slope) can hug that curve — a line has only two knobs and both control straightness, neither controls bend.

PICTURE.

The red line is the best possible straight line, and it still misses the curved dots at both ends. We need a knob for curvature.

---

Step 2 — The one idea: invent a new axis that is $x^2$

WHAT. We take each input $x$ and manufacture a brand-new number from it: $x^2$ (that is $x$ times itself). We now pretend this is a second, independent feature. So a point that used to be described by one number $x$ is now described by two numbers: $(x,x^2)$.

WHY. Here is the whole trick. Our line-fitter only knows how to draw flat things. But a flat plane living in a higher-dimensional space can look curved when we shadow it back down. By adding the $x^2$ axis, we lift the curved data up into a space where it becomes straight again — and then the flat fitter can handle it.

PICTURE.

On the left: the curved data in the ordinary $(x,y)$ plane. On the right: the same points lifted onto the new $(x^2,y)$ axis — notice they now line up straight! The curvature was hidden in the $x^2$ direction all along.

$\underset{\text{old feature}}{\underbrace{x}}⟶\underset{\text{engineered feature vector }\varphi (x)}{\underbrace{(1,x,x^2)}}$

• The $1$ ::: a constant column, so the fit still has an intercept knob ${\theta }_0$.
• The $x$ ::: keeps the ordinary slope information.
• The $x^2$ ::: the new knob — it lets the fit bend.

This manufacturing of features is feature engineering's twin idea.

---

Step 3 — The model: still linear, just wearing a curve costume

WHAT. Our prediction rule is now

$$h_{\theta }(x)={\theta }_0\cdot 1+{\theta }_1\cdot x+{\theta }_2\cdot x^2.$$

WHY — read every symbol.

• $h_{\theta }(x)$ ::: the model's guess for $y$ at input $x$ (the little $\theta$ subscript means "using knob settings $\theta$").
• ${\theta }_0$ ::: height knob (multiplies the constant $1$).
• ${\theta }_1$ ::: straight-slope knob (multiplies $x$).
• ${\theta }_2$ ::: bend knob (multiplies $x^2$) — positive bends the curve into a smile, negative into a frown.

The crucial observation: the knobs ${\theta }_0,{\theta }_1,{\theta }_2$ enter only by multiplication and addition — never as ${\theta }^2$ or $\sin \theta$. That is exactly what "linear in the parameters" means, and it is why our old linear-regression machinery still applies untouched.

PICTURE.

Watch the same curve morph as we turn the ${\theta }_2$ bend knob from negative (frown) through zero (flat line) to positive (smile).

---

Step 4 — Stack every example: the design matrix $\Phi$

WHAT. We have $m$ training points (say $m=4$). For each point we build its feature row $[1,x,x^2]$ and stack all rows into one grid called the design matrix $\Phi$:

$$\Phi =\left[\begin{array}{ccc}1& x^{(1)}& (x^{(1)}{)}^2\\ 1& x^{(2)}& (x^{(2)}{)}^2\\ 1& x^{(3)}& (x^{(3)}{)}^2\\ 1& x^{(4)}& (x^{(4)}{)}^2\end{array}\right],\mathbf{y}=\left[\begin{array}{c}{y}^{(1)}\\ y^{(2)}\\ y^{(3)}\\ y^{(4)}\end{array}\right].$$

WHY the shape.

• Each row ::: one training example, wearing all its manufactured features.
• Column 1 (all $1$s) ::: feeds the intercept ${\theta }_0$.
• Column 2 ($x$ values) ::: feeds the slope ${\theta }_1$.
• Column 3 ($x^2$ values) ::: feeds the bend ${\theta }_2$.
• The superscript $(i)$ ::: labels which example (not a power!). So $x^{(2)}$ is the 2nd point's input, and $(x^{(2)}{)}^2$ is that input squared.

PICTURE.

Each colored dot on the left becomes one colored row on the right. The grid is just the data, re-dressed.

---

Step 5 — Measure wrongness: the cost surface

WHAT. For a given setting of knobs, the model's total error is

$$J(\theta )=\frac{1}{2m}\sum _{i=1}^m(h_{\theta }(x^{(i)})-y^{(i)}{)}^2.$$

WHY every piece.

• $h_{\theta }(x^{(i)})-y^{(i)}$ ::: the residual — how far the guess sits from the truth for point $i$.
• The square ::: makes over- and under-shoots both count as positive, and punishes big misses more. It also makes the surface a smooth bowl (see picture), so calculus can find the bottom.
• $\sum$ ::: add the squared misses over all $m$ points.
• $\frac{1}{2m}$ ::: average ($\frac{1}{m}$) and a friendly $\frac{1}{2}$ so the derivative later comes out clean.

PICTURE.

$J(\theta )$ is a bowl over the knob-space. The lowest point of the bowl is the best fit. Our whole job is to find that bottom.

---

Step 6 — Roll to the bottom: the normal equation

WHAT. The bottom of a smooth bowl is exactly where the ground is flat — where the gradient (the vector of slopes of $J$ in each knob direction) is zero. Setting ${\nabla }_{\theta }J=0$ gives:

$${\Phi }^T\Phi \theta ={\Phi }^T\mathbf{y}⟹\boxed{\theta =({\Phi }^T\Phi {)}^{-1}{\Phi }^T\mathbf{y}}$$

WHY each symbol.

• ${\nabla }_{\theta }J$ ::: "which way is uphill, and how steep" for every knob at once; zero means we're at the flat bottom.
• ${\Phi }^T$ ::: $\Phi$ flipped so rows become columns — needed to line up the sums.
• ${\Phi }^T\Phi$ ::: a small square grid (here $3\times 3$) summarizing how the features overlap.
• $({\Phi }^T\Phi {)}^{-1}$ ::: the inverse — the matrix "undo" button that isolates $\theta$.
• ${\Phi }^T\mathbf{y}$ ::: how each feature correlates with the target.

This is word-for-word the linear-regression normal equation. Only $\Phi$ changed — the algebra never noticed we bent the curve. (Steepest-descent alternatives live in gradient descent variants.)

PICTURE.

The gradient arrows all point downhill; the star marks where they vanish — the solved $\theta$.

---

Step 7 — Edge case A: features of wildly different size

WHAT. If $x$ ranges up to $100$, then $x^2$ reaches $10000$ and $x^3$ reaches $1000000$. The columns of $\Phi$ now span millions to ones.

WHY it breaks. The bowl $J(\theta )$ becomes a long, thin, tilted canyon instead of a round bowl. Gradient descent zig-zags forever, and $({\Phi }^T\Phi {)}^{-1}$ becomes numerically explosive (nearly non-invertible). The fix is feature scaling: after manufacturing each feature, replace it with

$$x_j\leftarrow \frac{x_j-{\mu }_j}{{\sigma }_j},$$

where ${\mu }_j$ is that feature's mean and ${\sigma }_j$ its spread. This re-rounds the canyon into a bowl.

PICTURE.

Left: the stretched canyon (unscaled) — descent ricochets. Right: the round bowl (scaled) — descent walks straight in.

---

Step 8 — Edge case B: too many knobs (overfitting)

WHAT. Raise the degree so the number of knobs $d+1$ meets or beats the number of points $m$. Now the curve can thread every dot exactly.

WHY it's a trap. Zero training error looks perfect, but the curve wiggles violently between points, chasing noise instead of the trend. Test error explodes. This is the bias–variance tradeoff: high degree kills bias but detonates variance. The cure is picking degree with a validation set, or reining knobs in with regularization.

PICTURE.

Degree 2 (green) rides the trend. Degree 8 (red) touches every dot yet lurches wildly at the edges — memorizing, not learning.

---

The one-picture summary

Curved data $\to$ lift with $x^2$ $\to$ stack into $\Phi$ $\to$ slide to the bottom of the cost bowl $\to$ read off $\theta$ $\to$ project the flat fit back down as a curve.

Recall Feynman retelling — say it back in plain words

A straight line has two knobs and both make it straight, so it can never bend to hug curved dots. The clever move: for each input $x$, secretly invent a companion number $x^2$ and treat it as a whole new direction. In that taller space the curved data straightens out, so our ordinary straight-line fitter works perfectly — we never rebuilt the fitter. We pile every example's features into a grid $\Phi$, measure total squared miss as a bowl-shaped cost, and roll to the bottom of the bowl with one clean formula $\theta =({\Phi }^T\Phi {)}^{-1}{\Phi }^T\mathbf{y}$. Two dangers: if $x^2$ dwarfs $x$, the bowl becomes a nasty canyon (scale the features to fix it); and if we add too many knobs, the curve memorizes the dots instead of the pattern (choose degree with validation). Fit a flat thing in a lifted space, and its shadow back home is a curve.

Recall

Why is polynomial regression still "linear"? ::: It is linear in the parameters $\theta$ (they only multiply and add), even though it is curved in $x$. What single object changes versus plain linear regression? ::: Only the design matrix $\Phi$ (now holding powers of $x$); the cost and normal equation are identical. What two dangers get their own edge-case steps? ::: Wildly different feature scales (fix: scaling) and too-high degree (fix: validation / regularization).