2.2.13 · D5Linear & Logistic Regression
Question bank — L1 (Lasso) regularization
The symbols used here, all defined in the parent note: is the weight (multiplier) on feature ; is the number of features (how many there are); is the number of training examples (rows of data); is the regularization strength; is the L1 norm (sum of absolute values); is the OLS coefficient (the value would take with no penalty).
The next figure shows the geometric "why": the L1 diamond (blue) has sharp corners on the axes, so the elliptical MSE contours (gray) first touch it at a corner, where one coordinate is exactly . Contrast the L2 circle (green), which is smooth everywhere and gets touched at a generic point with no zeros.
True or false — justify
A soft-threshold with changes the OLS solution
False — , so with zero penalty Lasso reproduces OLS exactly and nothing is shrunk or zeroed.
Increasing can only shrink weights, never flip their sign
True — soft-thresholding pulls each coefficient toward zero and clamps at zero; it never overshoots past zero, so a positive never becomes a negative .
Lasso always selects fewer features than there are training examples
True in the classic result — for Lasso selects at most nonzero features; see the geometric reason in the next question.
Why can Lasso keep at most nonzero features when ?
Each nonzero coefficient must satisfy an equation tying it to the data's fit; with only rows there are effectively independent such equations, so no more than coordinates can be simultaneously "active" (nonzero). Elastic Net relaxes this by adding L2 curvature that can activate more.
If two features are identical copies, Lasso splits the weight evenly between them
False in general — Lasso tends to arbitrarily keep one and zero the other (the solution is not unique), which is exactly the instability Elastic Net fixes by adding an L2 term.
Doubling every feature's measurement units leaves the set of selected features unchanged
False — the L1 penalty is not scale-invariant, so rescaling a feature changes its effective penalty and can change whether it survives. This is why you standardize first.
At the optimum, every nonzero weight has been shrunk by exactly relative to its OLS value
True in the orthonormal case only — there; with correlated features the shrinkage is coupled across coordinates and no longer a clean constant offset.
The Lasso cost function is non-convex because has a kink
False — is convex (a kink is fine; convexity forbids dips, not corners), so the whole Lasso objective is convex and has no spurious local minima.
Lasso can produce a model with all weights zero
True — for large enough that every , all penalized weights clamp to zero and the model just predicts the mean (the bias term).
Spot the error
" is differentiable everywhere, so we can just set its derivative to and solve."
The error: is not differentiable at — that kink is the whole point. You must handle as a separate case using the subgradient, which is what creates exact zeros.
"Ridge also gives sparsity if you make big enough."
The error: Ridge's solution approaches zero but never equals zero for finite (unless ). Its circular constraint region has no corners on the axes, so no exact zeros.
"We should penalize the bias term the same as the others for consistency."
The error: the bias sets the output's baseline level and carries no per-feature information; penalizing it would bias predictions toward zero for no benefit, so it is conventionally left unpenalized.
"Because Lasso is convex, plain gradient descent lands cleanly on the zeros."
The error: gradient descent uses the derivative, which is undefined at the kink; it oscillates around zero and rarely lands exactly on it. Use Coordinate Descent or Proximal Gradient (ISTA/FISTA) which respect the kink.
"A feature with a tiny OLS coefficient must be useless, so Lasso is safe to trust for causal claims."
The error: a small coefficient can mean a small scale, collinearity, or a genuinely weak-but-real effect. Lasso selects for prediction, not causation, and dropping a feature is not evidence it has no effect.
"Higher always improves generalization since simpler models overfit less."
The error: past the sweet spot, zeros out useful features and the model underfits (high bias). The right trades bias against variance — tune it with Cross-Validation. See Bias-Variance Tradeoff.
"The L1 diamond has corners, so Lasso solutions always sit exactly on an axis."
The error: contours can touch a diamond on an edge too (giving no zeros) — corners are just more likely to be hit because the ellipse aligns with them. Sparsity is probable, not guaranteed for every dataset.
Why questions
Why does the fixed per-feature "tax" of L1 (rather than a squared cost) cause dropping rather than shrinking?
Because the penalty's slope near zero is a constant , not vanishing — so a feature must "pay" per unit even for tiny weights, and it's cheaper to set it to exactly zero than to keep a sliver. L2's slope at the origin, so it never fully lets go.
Why do we study the orthonormal (decoupled) case to understand Lasso at all?
With orthonormal features each coordinate optimizes independently, so the whole problem separates into one-dimensional soft-thresholds — exposing the sparsity mechanism cleanly before the coupled correlated case complicates it.
Why does the same L1 penalty give sparse logistic classifiers, not just linear regressors?
The L1 term is independent of the loss; it plants a corner at regardless of whether the fit term is MSE or logistic cross-entropy. See Logistic Regression.
Why must features be standardized before fitting, not after?
The penalty acts on the coefficients during optimization, so their relative scales decide who gets penalized more; standardizing afterward cannot undo a selection already biased by scale.
Why is Lasso's selection among correlated features called "unstable"?
Tiny changes in the data or in can flip which one of a correlated group survives, because they are near-interchangeable for the fit — so the chosen feature is somewhat arbitrary and not reproducible.
Why does Ridge handle correlated groups more gracefully than Lasso?
Its smooth squared penalty spreads weight across correlated features rather than forcing an all-or-nothing choice, keeping the group together — the intuition Elastic Net borrows.
Why can cross-validation, not the training error, pick ?
Training error decreases monotonically as (toward OLS), so it would always pick zero regularization; cross-validation estimates generalization, which has a genuine minimum at an intermediate .
Edge cases
What does Lasso do when exactly (an OLS coefficient already zero)?
since , so it stays zero for any — the penalty never adds weight to a feature the fit didn't want.
What happens at the exact boundary ?
, so the coefficient is clamped to exactly zero — the boundary belongs to the "dropped" case.
As , what model does Lasso converge to?
Every penalized weight is driven to zero, leaving only the unpenalized bias, so the model predicts the constant mean of — maximum bias, minimum variance.
As , does Lasso equal OLS?
Yes, in the limit — the penalty vanishes and every soft-threshold reduces to the identity, so the solution converges to the ordinary least squares fit.
If a feature is a constant column (zero variance), what does Lasso do with its weight?
A zero-variance feature carries no information, its OLS coefficient is undefined/uninformative, and the L1 penalty will drive its weight to zero — a reason to drop constant columns before fitting.
With more features than samples (), can Lasso still produce a unique solution?
Not always — the fit term can be flat along directions the data can't distinguish, giving multiple L1-optimal solutions; this is another case where Elastic Net's strictly convex L2 piece restores uniqueness.
If two features are perfectly anti-correlated, how does Lasso behave?
It faces the same interchangeability problem as with positive correlation — it tends to keep one and zero the other, since either can proxy for the pair, making the choice unstable.
Recall One-line summary of every trap here
L1 drops features because its penalty has a constant slope into zero (a corner), which makes exact zeros likely but not guaranteed, sensitive to feature scale, unstable among correlated features, non-differentiable at zero, and requiring cross-validation to balance bias against variance.