Foundations — Polynomial regression

This page assumes you have seen nothing. Every symbol in the parent note is unpacked below, in an order where each piece rests on the one before it. When you finish, re-reading the parent will feel like reading your own handwriting.

0. The picture we keep returning to

Look at the scattered dots. A straight line (blue) is honest but too stiff — it cannot bend to follow the smile-shaped cloud. The pale-yellow curve can bend. Every symbol we define exists to describe, build, or bend that yellow curve.

1. A data point: $(x,y)$

Picture: one dot on a graph. Its horizontal position is $x$, its vertical position is $y$. That is the entire meaning — a dot is a pair of numbers.

Why the topic needs it: regression is the craft of guessing $y$ from $x$. Without a pair, there is nothing to fit.

2. Many data points: the superscript $x^{(i)}$

The parent writes $x^{(i)}$ and $y^{(i)}$ with a number in parentheses up high. This is not a power.

Picture: number your dots left to right. Dot 3 has coordinates $(x^{(3)},y^{(3)})$.

Why the topic needs it: we fit to all the dots at once, so we must be able to point at each one.

3. Multiplying a number by itself: powers $x^2,x^3,x^d$

Picture: if $x$ is the side of a square, $x^2$ is its area; if $x$ is the edge of a cube, $x^3$ is its volume. Bigger exponent ⇒ the number grows faster as $x$ grows.

Notice in the figure how $x^3$ (pink) shoots up far steeper than $x$ (white). This steepness gap is exactly why feature scaling becomes non-negotiable later.

Why the topic needs it: curves are built by mixing different powers of $x$. The bend comes from $x^2$ and beyond.

4. The polynomial: ${\theta }_0+{\theta }_{1}x+{\theta }_2{x}^2+\cdots$

Read it left to right:

• ${\theta }_0$ — a plain number, added no matter what $x$ is (it lifts the whole curve up/down).
• ${\theta }_{1}x$ — the straight-line part.
• ${\theta }_2{x}^2$ — the first bend.
• ...and so on up to degree $d$.

Picture: each term is a "voice" in a choir. ${\theta }_2$ controls how loudly the $x^2$-voice sings; turn it up and the curve bends harder. Silence every voice above ${\theta }_1$ and you are back to a straight line.

Why the topic needs it: this sum is the shape of the yellow curve from figure 1.

5. The parameters: $\theta$ (theta)

Picture: a row of volume sliders on a mixing desk. Training = finding the slider positions that make the curve hug the dots best.

Why the topic needs it: the $\theta$s are the answer we solve for.

6. The hypothesis: $h_{\theta }(x)$

Picture: feed $x$ into the machine, out comes a height $h_{\theta }(x)$ — that is where the yellow curve sits above that $x$.

Why the topic needs it: it is the curve, written as a function. We compare $h_{\theta }(x^{(i)})$ against the real $y^{(i)}$ to see how wrong we are.

7. Turning one input into many: the feature map $\varphi (x)$

The parent writes $\varphi (\mathbf{x})=[1,x,x^2,\dots ,x^d]$. Here $\varphi$ (Greek "phi") is a recipe.

Picture: one seed ($x$) going into a machine that spits out several fruits ($1,x,x^2,\dots$). We then treat every fruit as an ordinary, independent input.

Why the topic needs it: this is the whole trick. After $\varphi$, the curved problem looks like a flat multi-feature linear problem — solvable by tools you already own.

8. Vectors, the dot product, and ${\theta }^T\varphi (x)$

Picture: two rows of boxes side by side; pair them up, multiply each pair, drop all the products into one bucket and sum. That single sum is the dot product.

Why the topic needs it: it is a compact way to write the whole polynomial. $h_{\theta }(x)={\theta }^T\varphi (x)$ says "dot the knobs with the manufactured features."

9. Stacking every example: the design matrix $\Phi$

The figure colour-codes it: each row (blue band) is one dot's full feature recipe $\varphi (x^{(i)})$; each column (pink band) is one feature across all dots.

Why the topic needs it: the normal equation $\theta =({\Phi }^T\Phi {)}^{-1}{\Phi }^T\mathbf{y}$ needs every example laid out at once. $\Phi$ is that layout.

10. Measuring wrongness: the cost $J(\theta )$

Picture: for each dot, draw a vertical rubber band from the dot to the curve; its length is the gap. $J$ is (half the average of) the squared band lengths. Training = shortening the bands.

Why the topic needs it: we cannot say "hug the dots" to a computer. $J$ turns "hug the dots" into a number to minimise.

11. The knob that solves it: normal equation

The parent's $\theta =({\Phi }^T\Phi {)}^{-1}{\Phi }^T\mathbf{y}$ uses two new pieces:

• ${\Phi }^T\Phi$ — a small square grid built from the data (see the worked example in the parent).
• $(\cdot {)}^{-1}$ — the inverse, the matrix version of "divide by."

Why it can go wrong: if features differ wildly in scale (recall figure 2), ${\Phi }^T\Phi$ becomes nearly impossible to invert — this is the numerical instability the parent warns about, and the reason for feature scaling.

How it all fits together

Every arrow says "you need the tail before you can understand the head." The whole graph pours into Polynomial regression.

Where these foundations reappear

• The steepness gap of high powers (§3) forces Feature scaling.
• Choosing degree $d$ (§3) without overfitting is the bias–variance tradeoff, controlled by regularization and judged on a validation split.
• Minimising $J$ (§10) when $\Phi$ is huge is done by gradient descent variants instead of the inverse.

Equipment checklist

Cover the right side; can you answer before revealing?