Polynomial regression: Question bank — AI-ML deep dive

True or false — justify

Polynomial regression is a non-linear learning algorithm.

False — it is ordinary linear regression run on transformed features; "linear" means linear in the parameters

θ

, and

x, x^{2}, x^{3}

are just fixed known numbers per example.

You need a brand-new optimizer (not gradient descent or the normal equation) to fit a polynomial.

False — because the model is still linear in

θ

, the exact same normal equation

θ = (Φ^{T} Φ)^{- 1} Φ^{T} y

and the same gradient descent apply unchanged.

A degree-

d

polynomial always fits training data at least as well as a degree-

(d - 1)

one.

True — the lower-degree model is a special case (set the top coefficient to

0

), so the higher-degree fit can never have larger training error, only equal or smaller.

Lower training error means the model will do better on new data.

False — a high-degree model can drive training error to zero by memorizing noise, yet fail badly on unseen points; that gap is exactly the overfitting problem.

Feature scaling for polynomial features is optional and only affects speed.

False — with raw features

x^{2}

and

x^{3}

span orders of magnitude, making

Φ^{T} Φ

ill-conditioned; scaling can be the difference between a stable solution and a numerically garbage one.

The intercept column of

1

's in

Φ

is just a formality with no effect.

False — it lets the curve shift vertically (

θ_{0}

); without it every polynomial would be forced through the origin, badly biasing the fit.

Adding an

x_{1} x_{2}

interaction term keeps the model linear in the parameters.

True —

x_{1} x_{2}

is a single precomputed number per example, so its coefficient enters linearly; it is just one more column in

Φ

If a degree-2 fit passes exactly through all points, you should always prefer it over a straight line.

False — an exact fit on few points may be coincidence or overfit; you judge generalization on held-out data via validation, not on training error.

Spot the error

"We fit

h (x) = θ_{0} + θ_{1} x^{2}

, so this is a non-linear model that gradient descent can't handle."

The error: dropping the linear term doesn't make it non-linear in

θ

;

x^{2}

is a known feature and gradient descent handles it fine.

"With 5 data points and a degree-4 polynomial, zero training error proves we found the true pattern."

The error: 5 parameters for 5 points guarantees an exact fit by construction, so zero error proves nothing about the underlying pattern — it is a red flag for overfitting.

"I scaled the original

x

before creating

x^{2}, x^{3}

, so scaling is done."

The error: scaling before transforming still leaves

x^{2}, x^{3}

on different scales; you must scale after generating the polynomial features so each column has comparable magnitude.

"Higher degree lowers bias, so higher degree is always the safer choice."

The error: lowering bias raises variance (bias-variance tradeoff); pushing degree up eventually makes test error worse, not better.

Φ^{T} Φ

has

d + 1

rows because we have

d + 1

training examples."

The error:

Φ^{T} Φ

(d + 1) \times (d + 1)

because there are

d + 1

features (columns of

Φ

); the number of examples

m

is the number of rows of

Φ

, which cancels out in the product.

"To model two features up to degree 2, I only need

x_{1}^{2}

and

x_{2}^{2}

The error: you also need the interaction term

x_{1} x_{2}

(and the linear terms) to capture how the two features jointly influence the target.

Why questions

Why does calling it "linear" regression not contradict fitting a curved parabola?

Because the curvature comes from the features (

x^{2}

), while the fitted quantity is a straight-line combination of coefficients

θ

— linearity refers to the parameters, not the shape of the output curve.

Why does the normal equation become numerically unstable without scaling?

Widely different feature scales make columns of

Φ

nearly dependent in floating point, so

Φ^{T} Φ

is near-singular and inverting it amplifies rounding errors enormously.

Why do we choose degree with a validation set instead of training error?

Training error only ever decreases (or stays flat) as degree rises, so it can't detect overfitting; validation error rises once the model starts memorizing noise, revealing the sweet spot.

Why can two features with an interaction term outperform two separate polynomial terms?

Because interactions capture joint effects (e.g. rainfall and heat together), which no sum of single-variable terms can represent — the effect of one feature depends on the value of the other.

Why does the derivation of

θ = (Φ^{T} Φ)^{- 1} Φ^{T} y

look identical to plain linear regression?

Because once features are transformed, the least-squares problem is literally the same optimization in a new feature space; only the entries of

Φ

changed, not the algebra.

Why might gradient descent be preferred over the normal equation for very high-degree polynomials?

The normal equation costs inverting a

(d + 1) \times (d + 1)

matrix and is fragile when it is ill-conditioned; gradient descent avoids the explicit inverse and scales better in feature count.

Edge cases

What happens if the polynomial degree

d

satisfies

d + 1 > m

(more parameters than examples)?

The system is underdetermined — infinitely many coefficient sets give zero training error,

Φ^{T} Φ

is singular, and the model overfits catastrophically.

What is a degree-0 polynomial regression, geometrically?

It is

h (x) = θ_{0}

, a flat horizontal line at the mean of

y

— it ignores

x

entirely and represents maximum bias, minimum variance.

If all your

x

values are identical, can you fit any polynomial degree

\geq 1

No — every column beyond the intercept becomes a constant, so

Φ

has dependent columns,

Φ^{T} Φ

is singular, and no unique solution exists.

What does a degree-1 polynomial regression reduce to exactly?

Ordinary simple linear regression

h (x) = θ_{0} + θ_{1} x

— polynomial regression is a strict generalization that contains the straight-line model as its degree-1 case.

If you standardize a feature that happens to already have mean 0 and std 1, what changes?

Nothing meaningful — subtracting

0

and dividing by

1

leaves the values unchanged, so scaling is a safe no-op on already-standardized features.

What happens to the fitted curve far outside the range of your training

x

values?

High-degree polynomials shoot off to

\pm \infty

rapidly, so extrapolation beyond the data range is wildly unreliable regardless of how good the in-range fit looked.

Recall One-line self-check

Can you state, without looking, why "linear" regression can fit a parabola? ::: The output curves because features like $x^{2}$ are curved, but the model is a straight-line (linear) combination of the parameters $θ$ .