5.6.1 · D5Machine Learning (Aerospace Applications)
Question bank — Linear regression — normal equation, gradient descent derivation
This bank hunts the misunderstandings that cause real bugs: sign errors in the gradient, thinking the normal equation always works, confusing what does. Each item is a one-line reveal — cover the answer, commit to a guess, then check. Every answer explains the reasoning, not just the verdict.
Parent: Linear Regression (parent topic).
True or false — justify
Squaring the errors is just a convenience; we could use absolute value and get the same line.
False. Squared error is differentiable everywhere and gives one smooth quadratic bowl with a unique minimum; absolute error has a kink at zero and generally yields a different line (the median-like fit), not the same one.
The cost is convex, so any local minimum is the global minimum.
True. is a quadratic bowl (its Hessian is positive semi-definite), so it has no separate local minima to get trapped in — see Convex Optimization.
The normal equation always returns a valid answer for any dataset.
False. It requires to be invertible; with perfectly correlated features (multicollinearity) or fewer samples than features, is singular and the plain inverse does not exist — you need the Matrix Pseudoinverse.
Gradient descent and the normal equation converge to different answers on the same data.
False. Both minimize the same convex , so (given a healthy and enough iterations) they land on the same . They differ in how they get there, not where.
The factor in the cost changes which is optimal.
False. Multiplying by any positive constant scales the whole bowl vertically but does not move its lowest point; the minimizing is unchanged. The constant only rescales the gradient (and so effectively rescales ).
At the optimum, the error vector is perpendicular to every column of .
True. The normal equation literally says the residual has zero dot product with each feature column — the geometric heart of Least Squares Estimation.
A larger learning rate always makes gradient descent reach the minimum faster.
False. Too-large overshoots the minimum each step, so the cost oscillates or diverges. There is a sweet spot; beyond it, bigger is slower or fatal.
Stochastic gradient descent (SGD) follows the exact same path as full-batch gradient descent, just faster.
False. SGD uses one sample's noisy gradient per step, so its path zig-zags around the true descent direction; it is faster per step but jitters near the minimum instead of settling exactly.
Spot the error
A student writes the update as .
The sign is wrong — it should be minus. Adding the gradient walks uphill, increasing the cost every step; you must step against the gradient to descend.
Someone computes the gradient as and then subtracts it in the update.
Two wrongs don't cancel here — this residual is , the negative of the true error, so their "gradient" already has the wrong sign; subtracting it climbs uphill. Correct gradient uses .
A student says " is , so inverting it costs ."
The cost is , not — the matrix being inverted has size equal to the feature count , independent of the number of samples .
Code fits raw features where altitude is in the 10,000s and Mach number is near 1, using gradient descent, and it "won't converge."
The features are on wildly different scales, so the cost bowl is a stretched ravine and a single can't suit both directions. Apply Feature Scaling first; the normal equation would be immune to this since it solves in one shot.
A student writes the pseudoinverse solution but forgets to add a column of ones to .
Without the ones column there is no bias term, forcing the fitted line through the origin. Any data whose best line has nonzero intercept will be fit badly.
"I'll just use SGD's update with the same I used for full-batch GD."
SGD gradients are noisier and typically need a smaller (often decaying) ; reusing the batch often makes SGD bounce around and never settle.
Why questions
Why do we transpose (giving ) inside the gradient?
Because the gradient must be a -vector, one number per weight; maps the -dimensional error vector back into feature space, summing each feature's contribution to the total error.
Why is allowed to equal in the derivation?
Both are scalars, and a scalar equals its own transpose; taking the transpose of reverses the product to , which is therefore the same number.
Why does setting the gradient to zero find a minimum and not a maximum here?
Because is a convex bowl (positive semi-definite Hessian ), so its only stationary point curves upward in every direction — a minimum, never a maximum or saddle.
Why prefer gradient descent over the normal equation when is large (e.g. aerospace sensor fusion with thousands of features)?
The normal equation's matrix inversion becomes prohibitively expensive; each gradient step is only , making iteration far cheaper for high-dimensional problems.
Why does adding a regularization term (Ridge) rescue a singular ?
Ridge adds , giving which is always invertible for ; it also shrinks weights to curb overfitting — see Regularization (Ridge, Lasso).
Why does the geometric picture call a "projection" of ?
can only reach points in the column space of ; the closest such point to is the foot of the perpendicular dropped from onto that space — exactly an orthogonal projection.
Why do we monitor instead of just running a fixed number of iterations?
A flattening cost signals we're already near the bottom, so we can stop early and save computation instead of wastefully iterating in a region where barely changes.
Edge cases
What does the normal equation give if all data points already lie exactly on one line?
It returns that line exactly, and the residual is the zero vector — the fit is perfect with cost .
What happens if you set in gradient descent?
The update becomes , so the weights never move; you stay stuck at the random initialization forever.
You have two identical feature columns (perfect duplication). What breaks and what still works?
is singular so the plain inverse fails, but the pseudoinverse or gradient descent still return a valid (though non-unique) minimizing — many weight combinations give the same predictions.
What is the fit when you have exactly one data point and one feature-plus-bias (, )?
The system is under-determined: infinitely many lines pass through a single point, so is singular and there is no unique solution — the pseudoinverse picks the minimum-norm one.
If every target value is the same constant , what does the fitted model look like?
The optimal line is flat: the weight(s) on the features go to zero and the bias becomes , so everywhere with zero residual.
As the number of gradient descent iterations with a valid small , what value does approach?
It converges to the exact normal-equation solution , since both minimize the same convex cost.
What does mini-batch gradient descent reduce to when the batch size equals the full sample count ?
It becomes ordinary full-batch gradient descent — the "batch" is the entire dataset, giving the exact (non-noisy) gradient each step.
Recall Quick self-test
The single most common sign bug in this whole topic is::: writing the update with instead of , or using where is needed — both climb uphill. The normal equation fails exactly when::: is singular (multicollinearity or ). Feature scaling matters for::: gradient descent (stretched ravines), not the normal equation.