Question bank — L2 (Ridge) regularization

False. Ordinary Least Squares minimizes training error alone, so nothing can beat it on training data; adding the $\lambda \Vert \mathbf{w}{\Vert }^2$ fee can only raise training error. Ridge wins on test error.

False (mostly). The penalized weights $w_1,\dots ,w_d$ shrink toward $0$, but the intercept $w_0$ is not penalized, so predictions collapse to the mean of $\mathbf{y}$, not to $0$.

True — the penalty term vanishes, so $J$ becomes the pure OLS cost and $(X^{\top }X+0\cdot I{)}^{-1}=(X^{\top }X{)}^{-1}$, provided $X^{\top }X$ is invertible.

True. OLS fails because $X^{\top }X$ is singular, but $X^{\top }X+\lambda I$ with $\lambda >0$ is strictly positive definite and thus invertible — Ridge cures the ill-posed case.

False. Test error typically dips then rises (the U-shape in the middle figure): too little $\lambda$ overfits, too much underfits. There is a sweet spot found by Cross-Validation.

True (and it was already convex). OLS cost is convex; adding $\lambda \Vert \mathbf{w}{\Vert }^2$ (a convex bowl) keeps it convex and makes it strictly convex, so the minimum is unique.

False. L2 shrinks weights smoothly but almost never sets one exactly to $0$; that is the job of L1 (Lasso) regularization, whose diamond-shaped constraint has corners on the axes (left figure).

True. Because $\lambda \sum w_j^2$ penalizes every weight equally, unequal feature scales make the penalty unfair; standardization (Feature Scaling / Standardization) is what makes Ridge scale-invariant.

False. Since ${\sigma }_j^2\ge 0$ and $\lambda \ge 0$, the denominator is always $\ge$ the numerator, so the factor lies in $[0,1)$ for $\lambda >0$ and equals $1$ only when $\lambda =0$ — it only ever shrinks, never amplifies.

The weights are small, not zero — the circular constraint has no corners touching the axes (left figure). Use L1 (Lasso) regularization or Elastic Net for genuine sparsity.

Training MSE is monotonically increasing in $\lambda$, so it always votes $\lambda =0$. $\lambda$ is a hyperparameter and must be tuned on held-out data via Cross-Validation.

The intercept sets the output baseline; shrinking it biases predictions toward $0$ for no reason. Exclude $w_0$ from the penalty (or center $\mathbf{y}$ first).

Wrong direction — adding $\lambda I$ ($\lambda >0$) is precisely what makes $X^{\top }X+\lambda I$ invertible. Singularity is what Ridge is built to fix.

That giant-cancelling pattern is the OLS pathology. Ridge minimizes $w_1^2+w_2^2$, so it prefers small equal weights (e.g. $\approx 1$ each) that sum to the same signal — cheaper norm.

A feature in millimeters carries a $1000\times$ smaller weight, so its penalty contribution is tiny and unfair. Standardize first — the penalty does not fix scale mismatch by itself.

That is data leakage: the scaler saw the test set's mean and variance, so information bled into training. Fit the scaler on the training fold only, then apply those same statistics to validation/test.

Correlated features let OLS pick huge opposite weights that cancel on training data but explode on new data; the $\sum w_j^2$ fee makes such large-magnitude combinations expensive, forcing calmer weights.

The factor $\frac{{\sigma }_j^2}{{\sigma }_j^2+\lambda }$ is $\approx 1$ when ${\sigma }_j$ is large (well-measured, high-signal) and $\approx 0$ when ${\sigma }_j$ is small (noisy, unstable) — so weak, noisy directions are shrunk away (right figure).

Test error $\approx {\text{bias}}^2+\text{variance}+\text{noise}$; Ridge trades a tiny bias increase for a large variance drop, a net reduction — the core Bias-Variance Tradeoff.

Every choice of $\lambda$ in the penalty $\lambda \Vert \mathbf{w}{\Vert }^2$ corresponds to some budget $t$ in "minimize the fit subject to $\Vert \mathbf{w}{\Vert }^2\le t$." Geometrically (left figure): the best point sits where the elliptical loss contours first touch the constraint ball — and at that touch point, the two forces (pull-toward-OLS vs push-toward-origin) balance, which is exactly the stationary condition the $\lambda$ term encodes. A bigger $\lambda$ pushes harder, shrinking the ball to a smaller $t$.

$\Vert \mathbf{w}{\Vert }^2\le t$ is a sphere (smooth, no corners), so the tangency with the loss contours is generically off-axis → nonzero weights; L1's $\sum |w_j|\le t$ has corners on the axes → exact zeros (left figure contrasts both).

A smooth spherical constraint has no flat faces or corners resting on the coordinate axes, so the optimal tangency point almost never lands on an axis where a weight would be $0$.

It shrinks the weight toward $0$ but leaves a small nonzero value; only L1 (Lasso) regularization can eliminate it entirely.

Undefined — the inverse $(X^{\top }X{)}^{-1}$ does not exist. Any $\lambda >0$ restores a unique solution; $\lambda =0$ leaves us in the ill-posed OLS regime.

It splits the required signal equally between them (smallest-norm split), since $1^2+1^2$ beats any lopsided combination giving the same fit — it never blows up as OLS can.

Even here Ridge biases weights downward; if the data was not overfit, added $\lambda$ mostly hurts by underfitting — regularization is not free when there was little variance to cut.

No — the count stays at $d$ (all nonzero); only their magnitudes shrink. This is the sharpest behavioural contrast with Lasso, where the count drops.

Yes — Logistic Regression adds $\lambda \Vert \mathbf{w}{\Vert }^2$ to its cross-entropy loss for the identical shrinkage effect, though there is no closed form (solved iteratively).