4.5.41 · D5Linear Algebra (Full)
Question bank — Least squares — normal equations, QR approach
Before you start, the vocabulary you must have (all built in the parent note): the residual (the "miss" vector), the column space (all reachable outputs ), and the projection (the shadow of on that subspace). See Orthogonal Projection and Column Space and Rank if any of these feel shaky.
True or false — justify
Every claim here is either true or false. Say which, then give the one-sentence reason — that reason is the whole point.
The least-squares solution always makes exactly.
False — it makes the closest point to in ; equality holds only in the lucky case where already lives in the column space.
If lies in , the residual is the zero vector.
True — then is reachable, the projection lands exactly on , and ; least squares recovers an exact solution.
Minimising and minimising give the same .
True — squaring is a strictly increasing function of a non-negative length, so it preserves the location of the minimum while making the objective smooth and differentiable.
The projection matrix is invertible.
False — collapses everything orthogonal to to zero, so it is singular (unless is all of ); it satisfies , which for an invertible matrix would force .
For a tall full-rank , the normal equations always have exactly one solution.
True — full column rank makes invertible, so is unique.
The residual is orthogonal to .
False — is orthogonal to (hence to ), not to itself; is the leg-plus-leg of a right triangle, so in general.
QR factorization and normal equations can give genuinely different answers .
False — is algebraically identical to the normal equations; they differ only in floating-point roundoff, and QR is the more accurate route. See QR Factorization.
is symmetric for every matrix .
True — ; symmetry needs no assumption on , whereas invertibility needs full column rank.
Spot the error
Each line states a plausible-sounding claim or "solution step." Find the flaw and state the correct version.
"Since has no solution, just compute anyway."
is with , so it is non-square and has no inverse; you must project first, giving instead.
" is square and symmetric, therefore it is invertible."
Square and symmetric does not imply invertible — is invertible iff has independent columns (full column rank); dependent columns make it singular. See Column Space and Rank.
"The best fit makes the residual lie inside so it's consistent with the data."
Exactly backwards — the residual must be orthogonal to ; any component inside the column space could be cancelled by moving , so a minimising residual has zero in-space component.
"We solved but couldn't cancel , so QR fails."
is upper triangular and invertible (full rank), hence so is , so we legitimately cancel it to get ; QR does not fail.
"Forming is fine numerically because it's a small square matrix."
The size is not the issue — forming squares the condition number (), amplifying roundoff; QR avoids ever forming it. See Condition Number.
" is orthonormal, so is a square orthogonal matrix and ."
In the thin QR of a tall matrix, is with orthonormal columns, so but (it's the projection onto , not the identity).
"The projection is always shorter than , so strictly."
By Pythagoras , so with equality when (i.e. ); it is not always strict.
Why questions
Give the reason, not just the fact — this is where understanding lives.
Why do we square the residual length instead of minimising the length itself?
The plain length has a non-differentiable kink at zero, while the square is smooth everywhere; squaring keeps the same minimiser but lets calculus (set gradient to zero) produce clean linear normal equations.
Why must the residual be orthogonal to every column of , not just to ?
The columns span ; being orthogonal to all of them means orthogonal to the whole subspace, which is precisely the condition that no small move of can shrink the distance.
Why is the critical point of a minimum and not a maximum or saddle?
The Hessian is , and , so the Hessian is positive semidefinite — the surface curves upward in every direction, forcing a minimum.
Why does QR turn the least-squares solve into a back-substitution?
Substituting and using reduces the problem to , and upper triangular means each unknown is found by simple back-substitution from the bottom row up — fast and stable.
Why does the projection matrix satisfy ?
Projecting a vector already inside leaves it unchanged, so applying twice is the same as once; algebraically .
Why does Gram–Schmidt's "subtract the projection" step produce orthogonal columns?
Removing the component of along leaves exactly the part perpendicular to , so the new vector's dot product with is zero by construction. See Gram-Schmidt Process.
Why is least squares the natural tool for fitting a line to scattered data?
The data rarely lie exactly on any line (an overdetermined system), so we seek the line whose total squared vertical miss is smallest — precisely the least-squares projection of onto the two-column model space. See Linear Regression.
Edge cases
The scenarios the naive picture forgets — degenerate rank, non-uniqueness, and boundaries.
If has dependent columns, what happens to ?
is singular, so is non-unique — a whole family of solutions all give the same projection ; the Pseudoinverse (Moore-Penrose) picks the unique minimum-norm one.
Even when is non-unique, is the projection unique?
Yes — the closest point in a subspace is always unique; only the coordinates describing that point become ambiguous when the columns are dependent.
What is the least-squares solution when ?
, so (for full rank) and the residual is — the closest point to the origin in any subspace is the origin itself.
What if and is invertible (a square consistent system)?
Least squares degenerates to the ordinary solve: the residual is exactly zero, reduces to , and projection onto the full space is the identity.
What if is already orthogonal to all columns of ?
Then , giving , projection , and residual — the best fit is the zero vector because points entirely outside the model space.
If two data points share the same but different , can a line fit both exactly?
No — a function gives one per , so the system is inconsistent; least squares fits the line through their average height, and the residuals at that split the gap.
For a horizontal-line fit (single constant column of ones), what does least squares compute?
The projection onto the span of is the mean of the data — least squares of a constant model literally returns the average, the point minimising total squared deviation. See Linear Regression.
Does adding an extra data point ever increase the minimum residual norm?
Yes — a new point generally cannot be fit perfectly by the same low-dimensional model, so the total squared miss can only stay equal or grow; more constraints never make an overdetermined fit easier.
Recall One-line summary of the traps
Least squares = project, don't invert; the residual is orthogonal to (not inside) the column space; is invertible only with full column rank; and QR gives the same answer as the normal equations by a numerically safer road.