Elastic Net regularization

WHAT is it?

Definition Elastic Net

Elastic Net augments the least-squares loss with a convex combination of the L1 and L2 penalties. For weights $\mathbf{w}$ , data $(\mathbf{x}_i, y_i)$ : $J(\mathbf{w}) = \underbrace{\frac{1}{2n}\sum_{i=1}^{n}\left(y_i - \mathbf{w}^\top \mathbf{x}_i\right)^2}_{\text{data fit}} + \lambda\left( \underbrace{\alpha\lVert \mathbf{w}\rVert_1}_{\text{L1: sparsity}} + \underbrace{\tfrac{1-\alpha}{2}\lVert \mathbf{w}\rVert_2^2}_{\text{L2: shrink+group}} \right)$

$\lambda \ge 0$ = overall regularization strength.
$\alpha \in [0,1]$ = mixing ratio. $\alpha=1 \Rightarrow$ pure Lasso; $\alpha=0 \Rightarrow$ pure Ridge.

WHY combine them? (first-principles reasoning)

Why L1 gives sparsity: the L1 ball has corners on the axes. When the loss contour first touches this ball, it tends to touch at a corner → some coordinates become exactly $0$ .

Why L1 alone fails on correlated features: if $x_1 \approx x_2$ , the loss is nearly flat along the direction $w_1 + w_2 = \text{const}$ . L1's penalty $|w_1|+|w_2|$ is also constant along that direction (as long as signs match), so the solution is not unique — it wobbles arbitrarily.

Why adding L2 fixes it: the L2 penalty $w_1^2 + w_2^2$ is strictly convex and is minimized (for fixed sum) when $w_1 = w_2$ . So the L2 term breaks the tie and spreads the weight evenly across correlated features. This is the grouping effect.

HOW to derive the update (soft-thresholding)

We derive the coordinate-descent update for one weight $w_j$ . Assume features are standardized so $\frac{1}{n}\sum_i x_{ij}^2 = 1$ .

Step 1 — isolate $w_j$ . Fix all other weights. Define the partial residual (target minus everyone else's contribution): $r_i = y_i - \sum_{k\ne j} w_k x_{ik}.$ Why this step? Coordinate descent optimizes one variable at a time; everything not involving $w_j$ is a constant.

Step 2 — write the 1-D objective. $f(w_j) = \frac{1}{2n}\sum_i (r_i - w_j x_{ij})^2 + \lambda\alpha |w_j| + \frac{\lambda(1-\alpha)}{2}w_j^2$ Why this step? This is exactly $J$ with only $w_j$ free.

Step 3 — take the subgradient and set to 0. Let $\rho_j = \frac{1}{n}\sum_i x_{ij} r_i$ (correlation of feature $j$ with the partial residual). Using $\frac{1}{n}\sum x_{ij}^2 = 1$ : $0 \in -\rho_j + w_j + \lambda(1-\alpha)w_j + \lambda\alpha\,\partial|w_j|$ Why this step? The quadratic parts differentiate cleanly; $|w_j|$ has subgradient $\text{sign}(w_j)$ for $w_j\ne 0$ and $[-1,1]$ at $0$ .

Step 4 — solve, giving the soft-threshold. The minimizer is $\boxed{\,w_j = \frac{S(\rho_j,\ \lambda\alpha)}{1 + \lambda(1-\alpha)}\,}, \qquad S(z,\gamma)=\text{sign}(z)\,\max(|z|-\gamma,\ 0)$ Why this step? $S$ is the soft-thresholding operator: the L1 term shrinks $\rho_j$ toward zero by $\lambda\alpha$ (and kills it if $|\rho_j|\le\lambda\alpha$ ), while the L2 term divides by $1+\lambda(1-\alpha)$ , an extra proportional shrink.

Worked examples

Common mistake Steel-manned pitfalls

"Elastic Net is just Lasso with a tiny extra term — use $\alpha=0.99$ always." Why it feels right: you mostly want sparsity, so you keep $\alpha$ near 1. The fix: with correlated features, even a small L2 (say $\alpha=0.5$ – $0.8$ ) dramatically stabilizes selection. Tune $\alpha$ by cross-validation, not by gut.

" $\lambda$ and $\alpha$ do the same thing." Why it feels right: both are penalty knobs. The fix: $\lambda$ = how much total regularization; $\alpha$ = what flavor (L1 vs L2 mix). They are orthogonal — you tune a 2-D grid.

"Forgot to standardize features." Why it feels right: linear regression is scale-equivariant, so who cares. The fix: penalties are not scale-invariant. A feature in kilometers vs metres gets penalized differently. Always standardize before Elastic Net.

Active recall

Recall Forecast-then-Verify: predict before revealing

Q: If $\alpha=0$ , which method do you get, and does it produce zeros? A: Pure Ridge; it shrinks but never sets weights exactly to zero.

Q: Two perfectly correlated features — what does Elastic Net do with the weight? A: Splits it (roughly) equally between them (grouping effect), thanks to the L2 term.

Recall Feynman: explain to a 12-year-old

Imagine you're packing a backpack for a trip and you must pick which gadgets to bring. Lasso is a strict friend: "pick ONE flashlight, throw the other two away" — even if all three are equally good, and next time he might pick a different one at random. Ridge is a soft friend: "bring smaller versions of everything, nobody gets left home." Elastic Net is the wise friend: "throw out the truly useless junk (like Lasso), but for the three equally-good flashlights, just bring three small ones so it's fair and stable (like Ridge)." That fairness for equally-good items is why it doesn't panic when things look alike.

The 80/20 core

$J = \text{MSE} + \lambda(\alpha\|w\|_1 + \tfrac{1-\alpha}{2}\|w\|_2^2)$ .
Update = soft-threshold (numerator, L1) over $1+\lambda(1-\alpha)$ (denominator, L2).
L1 → sparsity; L2 → grouping/stability; Elastic Net → both.
Tune $\lambda$ (strength) and $\alpha$ (flavor) by CV; standardize first.

Connections

Ridge Regression — the $\alpha=0$ special case (L2 only).
Lasso Regression — the $\alpha=1$ special case (L1 only).
Soft-Thresholding Operator — the proximal operator powering the update.
Coordinate Descent — the algorithm used to solve it.
Bias-Variance Tradeoff — regularization trades variance for bias.
Feature Correlation & Multicollinearity — the problem Elastic Net specifically tames.
Cross-Validation — how $\lambda,\alpha$ are chosen.

What penalty does Elastic Net add to the loss?

A convex combination

\lambda(\alpha\|w\|_1 + \tfrac{1-\alpha}{2}\|w\|_2^2)

— both L1 and L2.

What does

\alpha

control in Elastic Net?

The mixing ratio between L1 and L2;

\alpha=1

is Lasso,

\alpha=0

is Ridge.

What does

\lambda

control?

The overall strength of regularization (independent of the L1/L2 flavor).

Why does Lasso struggle with correlated features?

Its objective is flat along the correlated direction, so selection is non-unique/unstable.

How does the L2 term fix correlated features?

L2 is strictly convex and minimized when equal weights, so it splits weight evenly (grouping effect).

Write the coordinate-descent update for

w_j

w_j = S(\rho_j,\lambda\alpha)/(1+\lambda(1-\alpha))

where

S

is soft-thresholding.

What is the soft-thresholding operator

S(z,\gamma)

\text{sign}(z)\max(|z|-\gamma,0)

— shrinks toward zero, sets to zero if

|z|\le\gamma

Which part of the update gives sparsity vs stability?

Numerator (L1 soft-threshold) → sparsity; denominator

1+\lambda(1-\alpha)

(L2) → stability/grouping.

Must you standardize features before Elastic Net?

Yes — penalties are not scale-invariant, so unstandardized scales bias which features get penalized.

\rho_j=0.3

\lambda=1

\alpha=0.5

, what is

w_j

0

, because

|0.3|<\lambda\alpha=0.5

so the soft-threshold is zero.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, Elastic Net basically Lasso (L1) aur Ridge (L2) ka combo hai. Problem yeh hai ki Lasso akela to features ko zero kar deta hai (sparsity milti hai, achhi baat), lekin jab do features aapas mein bahut correlated ho — jaise ek hi cheez do baar aa gayi — to Lasso confuse ho jaata hai aur randomly kisi ek ko rakh ke baaki ko phenk deta hai. Ridge iss situation ko handle kar leta hai, weight ko fairly baant deta hai, lekin Ridge kabhi kisi weight ko exactly zero nahi karta, so feature selection nahi hoti.

Elastic Net kehta hai "dono ka faayda le lo": loss mein $\alpha\|w\|_1 + \tfrac{1-\alpha}{2}\|w\|_2^2$ dono add kar do. Yahan $\lambda$ decide karta hai kitna regularization, aur $\alpha$ decide karta hai kaunsa flavor (α=1 pura Lasso, α=0 pura Ridge). Iska magic result hai grouping effect: agar do features ek jaise hain to L2 term unke beech weight ko barabar baant deta hai, aur L1 term bekaar features ko poori tarah zero kar deta hai. Best of both worlds.

Update rule bhi simple hai: $w_j = S(\rho_j,\lambda\alpha)/(1+\lambda(1-\alpha))$ . Upar wala soft-thresholding $S$ chhote correlation waale features ko maar deta hai (sparsity), aur neeche wala $1+\lambda(1-\alpha)$ baaki ko thoda dabaa deta hai (stability). Ek important baat — features ko pehle standardize zaroor karo, warna kilometre vs metre jaise scale differences se penalty galat lag jaayegi. Aur $\lambda$ , $\alpha$ dono ko cross-validation se tune karo, guess se nahi.

WHAT is it?

WHY combine them? (first-principles reasoning)

HOW to derive the update (soft-thresholding)

Worked examples

Active recall

The 80/20 core

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Linear & Logistic Regression

Connections