Elastic Net regularization: Question bank — AI-ML deep dive

Symbols used here (all built in the parent): $J$ = the loss we minimize; $w$ = the weight vector; $λ \geq 0$ = overall regularization strength; $α \in [0, 1]$ = mixing ratio (how much of the penalty is L1 vs L2); $∥ w ∥_{1} = \sum_{j} ∣ w_{j} ∣$ = sum of absolute weights; $∥ w ∥_{2}^{2} = \sum_{j} w_{j}^{2}$ = sum of squared weights; $ρ_{j}$ = correlation of feature $j$ with the partial residual; $S (z, γ) = sign (z) max (∣ z ∣ - γ, 0)$ = the soft-thresholding operator.

True or false — justify

If a claim is true, say why the mechanism forces it; if false, name the exact step that breaks.

True or false: Setting

α = 0

turns Elastic Net into Lasso.

False.

α = 0

kills the L1 term (

α ∥ w ∥_{1}

) and leaves only the L2 term, so you get pure Ridge — which never produces exact zeros.

α = 1

is Lasso.

True or false: Increasing

λ

makes the model sparser.

Partly false as stated. Raising

λ

increases total shrinkage, but sparsity comes from the L1 threshold

λ α

; if

α = 0

no amount of

λ

ever zeros a weight. Sparsity needs both

λ

and

α > 0

True or false: Because linear regression is scale-equivariant, standardizing features before Elastic Net is optional.

False. The penalty is not scale-invariant —

∣ w_{j} ∣

and

w_{j}^{2}

depend on the units of feature

j

. A feature in metres vs kilometres is penalized differently, so you must standardize first.

True or false:

λ

and

α

are just two names for the same "penalty knob."

False.

λ

sets how much total regularization;

α

sets what flavor (L1-vs-L2 mix). They are orthogonal, which is why you tune them on a 2-D grid.

True or false: With two perfectly correlated features, Lasso and Elastic Net give the same solution.

False. Lasso's objective is flat along the correlated direction, so it picks one champion arbitrarily; Elastic Net's strictly-convex L2 term uniquely prefers equal weights, splitting the load (the grouping effect).

True or false: Elastic Net can select more features than there are training examples (

p > n

True. Unlike pure Lasso (which saturates at

n

selected features), the L2 term lets Elastic Net keep whole correlated groups, so it can retain more than

n

nonzero weights.

True or false: The soft-thresholding operator

S (z, γ)

shrinks small

z

but leaves large

z

untouched.

False.

S

subtracts

γ

from every surviving coefficient's magnitude, not just small ones — large

∣ z ∣

still loses

γ

. It merely kills the ones with

∣ z ∣ \leq γ

entirely.

True or false: The L2 part of Elastic Net can, on its own, drive a weight to exactly zero.

False. The L2 term enters as division by

1 + λ (1 - α)

, a finite positive number — it shrinks proportionally but never reaches exact zero. Only the L1 threshold produces true zeros.

True or false: If

α = 0.99

, Elastic Net behaves essentially identically to Lasso in every situation.

False. Even a tiny L2 (

1 - α = 0.01

) restores strict convexity along correlated directions, dramatically stabilizing which feature gets selected. Small L2 has an outsized stabilizing effect.

Spot the error

Each statement contains one wrong claim. Find it and correct it.

"The numerator of the update,

S (ρ_{j}, λ α)

, is the L2 shrink and the denominator is the L1 shrink."

Reversed. The numerator (thresholding) is the L1 part → sparsity; the denominator

1 + λ (1 - α)

(proportional division) is the L2 part → grouping/stability.

"Elastic Net's penalty is

λ (α ∥ w ∥_{1} + (1 - α) ∥ w ∥_{2}^{2})

The L2 term carries a factor of

\frac{1}{2}

: it is

\frac{1 - α}{2} ∥ w ∥_{2}^{2}

. The half makes the derivative of the squared term clean (

w_{j}

instead of

2 w_{j}

), matching the update formula.

"Since L1 gives sparsity, using L1 alone with correlated features is fine — you just keep the best one."

The flaw is 'fine.' L1 alone keeps an arbitrary one and the choice is unstable across resamples, because the objective is flat along

w_{1} + w_{2} = const

. That instability, not the sparsity, is the problem.

"To make Elastic Net select fewer features, raise

α

toward 1 and lower

λ

Lowering

λ

reduces the threshold

λ α

, keeping more features. To select fewer, raise

α

and raise (or hold)

λ

so the L1 threshold grows.

"The grouping effect comes from L1 being strictly convex."

L1 is not strictly convex — it's flat along sign-matched directions, which is exactly why it fails on groups. The grouping effect comes from L2 being strictly convex and minimized at equal weights.

"Because

S (ρ_{j}, λ α)

appears, a feature with

ρ_{j} = 0

can still get a nonzero weight from the L2 term."

No. If

ρ_{j} = 0

then

∣ ρ_{j} ∣ \leq λ α

, so

S = 0

, so the numerator is zero and

w_{j} = 0

regardless of the L2 denominator. Zero numerator ⇒ zero weight.

Why questions

Answer with the mechanism, not a restatement.

Why does the L1 ball produce sparsity while the L2 ball does not?

The L1 ball has corners on the axes, so the loss contour tends to first touch it at a corner where some coordinates are exactly zero; the L2 ball is round with no corners, so contact points generically have all coordinates nonzero.

Why does adding L2 make the correlated-feature solution unique?

L2 is strictly convex, so

J

becomes strictly convex along the previously-flat direction

w_{1} + w_{2} = const

; a strictly convex function has exactly one minimizer, eliminating the wobble.

Why do we standardize the assumption

\frac{1}{n} \sum_{i} x_{ij}^{2} = 1

before deriving the update?

It removes the coefficient in front of

w_{j}

in the derivative, letting the quadratic term simplify to a clean

w_{j} + λ (1 - α) w_{j}

— otherwise each feature's shrink would depend on its own scale.

Why does the update divide by

1 + λ (1 - α)

instead of subtracting the L2 penalty?

The L2 term

\frac{λ ( 1 - α )}{2} w_{j}^{2}

differentiates to

λ (1 - α) w_{j}

, which is proportional to

w_{j}

; solving the linear equation moves it to the denominator, giving a multiplicative (proportional) shrink rather than a fixed subtraction.

Why can't we just average Ridge and Lasso solutions instead of blending their penalties?

The minimizer of a sum of penalties is not the average of the individual minimizers — optimization is nonlinear. Blending the penalties inside one convex objective is what yields the correct joint solution with both sparsity and grouping.

Why is cross-validation, not intuition, recommended for choosing

α

The best

α

depends on the unknown correlation structure and noise level of the data; CV empirically measures generalization across the

(λ, α)

grid rather than guessing which flavor mix helps.

Edge cases

Push the formula to its boundaries and say what happens.

Edge:

λ = 0

. What does Elastic Net reduce to?

The whole penalty vanishes and

J

becomes plain least-squares (OLS) — no shrinkage, no sparsity, and correlated features again cause instability/multicollinearity.

Edge:

α = 1

and

∣ ρ_{j} ∣ \leq λ

. What is

w_{j}

The threshold is

λ α = λ

, so

S (ρ_{j}, λ) = 0

and the denominator is

1 + λ \cdot 0 = 1

, giving

w_{j} = 0

— pure Lasso zeroing with no L2 shrink.

Edge: A feature perfectly uncorrelated with the residual (

ρ_{j} = 0

) under any

α > 0

S (0, λ α) = 0

, so

w_{j} = 0

; a feature with no relationship to what's left to explain is dropped immediately.

Edge: Two features that are negatively perfectly correlated (

x_{2} = - x_{1}

). Does grouping still split evenly?

Grouping still stabilizes, but the equal-magnitude split is

w_{1} = - w_{2}

(equal magnitudes, opposite signs) since that minimizes

w_{1}^{2} + w_{2}^{2}

for the required combined effect; L2 equalizes magnitude, respecting sign structure.

Edge:

α = 0

with a huge

λ

. What happens to all weights?

Pure Ridge with strong shrinkage: every weight is divided by a large

1 + λ

, so they all shrink toward (but never reach) zero — heavy bias, low variance, no selection.

Edge: A very large

ρ_{j}

that survives the threshold, as

α \to 0

The L1 subtraction

λ α \to 0

(almost no thresholding), while the L2 denominator

1 + λ (1 - α) \to 1 + λ

dominates — the weight is shrunk almost purely by proportional division, i.e. Ridge-like behavior.

Connections

Ridge Regression — the $α = 0$ boundary probed above.
Lasso Regression — the $α = 1$ boundary probed above.
Soft-Thresholding Operator — the $S (z, γ)$ at the heart of every "spot the error" here.
Coordinate Descent — where these one-coordinate updates live.
Feature Correlation & Multicollinearity — the correlated-feature traps target this.
Bias-Variance Tradeoff — the strong- $λ$ edge cases are pure bias-variance.
Cross-Validation — the answer to "why not intuition" for tuning.