2.2.3 · D5Linear & Logistic Regression
Question bank — Ordinary least squares derivation
This is a rapid-fire trap-detector for Ordinary Least Squares. Every item below targets a spot where intuition quietly lies. Cover the answer, commit to a reason (not just "true"/"false"), then reveal. If your reason is wrong even when your verdict is right, you have not understood the trap.
Prerequisites worth having open: Linear Regression Fundamentals, Gradient Descent, Ridge Regression, Maximum Likelihood Estimation, Pseudoinverse, QR Decomposition, Feature Scaling, Bias-Variance Tradeoff.
Symbols and pictures you need first
Before any trap makes sense, pin down the five objects every question below leans on. Say each one in plain words before you read on.
Look at the geometry that every "orthogonality" and "projection" answer secretly refers to:
The prediction vector can only ever live inside the flat plane . The closest point in that plane to the target is the foot of the perpendicular dropped from — the orthogonal projection. At that closest point the residual (red) is perpendicular to the whole plane, which is precisely .
True or false — justify
OLS always has exactly one solution.
False. Only when has full column rank is invertible; if columns are linearly dependent (rank ) the bowl has a flat valley floor, so infinitely many hit the same minimal cost.
The factor in the cost changes the optimal parameters.
False. Multiplying by any positive constant rescales the bowl's height but not the location of its lowest point, so is untouched — the factor is pure convenience.
Minimizing squared error is the same as minimizing absolute error.
False. Squaring punishes large residuals quadratically, so OLS is pulled harder toward outliers than absolute-error (L1) fitting, giving a different line.
At the OLS optimum, the residual vector is perpendicular to every column of .
True. The normal equation is exactly , meaning each column of has zero dot product with the residual — that orthogonality (see the projection figure) is why it is called the "normal" equation.
OLS assumes the noise in is Gaussian.
The derivation by least squares needs no such assumption, but OLS equals maximum likelihood estimation (Maximum Likelihood Estimation) precisely when the noise is Gaussian — so the Gaussian story is a justification, not a requirement.
Gradient descent and OLS can converge to different parameters on the same convex problem.
False provided the minimizer is unique. When has full column rank the bowl has one lowest point, so Gradient Descent — run with a step size small enough to guarantee descent — must land at the same ; if rank is deficient both methods only agree up to the flat valley.
Adding the column of ones to is optional if the data is already centered.
If both and have mean zero the fitted intercept is zero, so you can drop it — but for uncentered data omitting the ones column forces the line through the origin and biases the fit.
is always symmetric.
True. For any matrix, , so it equals its own transpose regardless of the data.
If is invertible, OLS is guaranteed to be numerically stable.
False. Invertibility is a yes/no fact; stability is a matter of degree. A near-singular (tiny determinant, huge condition number) inverts in theory but amplifies rounding errors badly — prefer QR Decomposition.
Ridge regression makes the residual orthogonal to just like OLS does.
False. Ridge solves , giving for . So the ridge residual is deliberately not orthogonal to the column space — we tilt away from the pure projection to shrink and cut variance (Ridge Regression).
Spot the error
" is , so inverting it costs ."
Wrong shape. is , so is — inversion is , driven by feature count , not sample count .
"Since , we cannot combine those two terms."
Error. Both are scalars, and a scalar equals its own transpose, so — that is exactly why the cross terms combine into .
"."
Missing factor of 2. For a symmetric matrix , ; here is symmetric, so the correct gradient carries the 2 that later cancels the .
"OLS is closed-form, so it is always faster than gradient descent."
False economics. The cost blows up for large ; for in the tens of thousands, iterative Gradient Descent is far cheaper in time and memory.
"With 5 samples and 20 features, OLS just needs a bigger matrix inversion."
No inversion exists. With the system is underdetermined, is singular, and infinitely many fit perfectly — you need regularization or the Pseudoinverse, not a bigger solver.
"Feature scaling doesn't matter for OLS because it has a closed form."
Half-true, dangerous. The exact solution is scale-equivariant, but poor scaling wrecks the condition number of , so Feature Scaling still improves numerical accuracy (and matters a lot for the gradient-descent path).
Why questions
Why does the residual being orthogonal to the column space mean we found the minimum?
Because the fitted vector is the orthogonal projection of onto (see figure); for any other in-space point , the vector splits into a piece perpendicular to the plane and a piece inside it, so by Pythagoras — the projection is strictly closest.
Why is the OLS minimum unique when has full column rank?
Compute the Hessian: . Full column rank means for any nonzero , , so the Hessian is positive definite. A positive-definite Hessian everywhere makes strictly convex — a strictly convex function has at most one minimizer, and the normal equation supplies it.
Why do we square residuals instead of taking their fourth power?
Squaring already fixes sign cancellation and stays differentiable everywhere; higher even powers would over-punish outliers and lose the clean linear normal equation, and they no longer correspond to Gaussian-noise likelihood.
Why does perfect correlation between two features break OLS?
Correlated columns are linearly dependent, so loses full column rank, , and the inverse ceases to exist — the model cannot tell how to split credit between the two identical directions.
Why does adding in Ridge always make the matrix invertible (for )?
is symmetric positive semi-definite (eigenvalues ); adding with shifts every eigenvalue up by , making them all strictly positive, hence positive definite and invertible.
Why does gradient descent need a bounded step size to reach ?
The gradient is Lipschitz with constant (the largest eigenvalue). A step size guarantees each update lowers ; larger steps overshoot the bowl and diverge — that is the precise meaning of "a good step size."
Why is the pseudoinverse the right tool when is singular?
The Pseudoinverse returns the minimum-norm solution among the infinitely many that minimize the cost, giving a unique, well-defined answer where the plain inverse fails to exist.
Why can OLS overfit even though it never "trains too long"?
Overfitting here is structural, not iterative: with many features relative to samples the model has enough freedom to fit the noise exactly, pushing it to the high-variance end of the Bias-Variance Tradeoff.
Edge cases
What does OLS return if all target values are identical?
It fits a flat line: the slope coefficients go to zero and the intercept equals that common value, since a constant already achieves zero residual.
What happens if a data row is duplicated many times?
Duplicates add no new column-rank, so stays invertible only if the distinct points already span the feature directions; the repeats simply reweight the loss toward that point, tilting the fit.
What is the OLS fit when there is exactly one data point per parameter (, points in general position)?
The system is square and (generically) full rank, so OLS interpolates every point with zero residual — a perfect fit that is usually pure overfitting, not good generalization.
What does OLS do with a feature column that is all zeros?
A zero column is trivially a linear combination of the others (coefficient zero), so loses full column rank and is singular; the corresponding is unidentifiable and must be dropped or regularized.
As two features become nearly (not exactly) collinear, what happens to the coefficients?
The condition number explodes, so the coefficients swing to huge, opposite-signed, unstable values — a warning sign that calls for Ridge or feature removal even though the inverse technically still exists.
What if you set the Ridge penalty ?
For ridge collapses back to plain OLS (no regularization). For you subtract from the eigenvalues of , which can make the matrix singular or indefinite — the "penalty" now rewards large , the problem may become unbounded below, and it is no longer a valid regularizer.
What if the noise is heavy-tailed rather than Gaussian?
OLS still minimizes squared error, but it is no longer the maximum-likelihood estimator; heavy tails (e.g. Laplace or Cauchy noise) make squared loss over-react to outliers, so robust losses (L1, Huber) or the corresponding MLE (Maximum Likelihood Estimation) fit better.
Recall Self-check before you leave
Which single geometric fact underlies the entire normal equation? ::: The residual is orthogonal to the column space of , i.e. — every other result is algebra flowing from this projection.