Visual walkthrough — Elastic Net regularization

We only need one idea from calculus and one picture-fact about slopes. We introduce each exactly when it is needed — nowhere earlier.

Step 1 — What are we even minimizing? Draw the bowl.

WHAT. We want the weight $w_j$ (the number multiplying feature $j$) that makes our cost as small as possible. "Cost" here means: how badly we predict, plus a penalty for using large weights.

Forget all the other weights for a moment — pretend they are frozen. Then the cost depends on one number $w_j$, so we can plot it as a curve: the horizontal axis is $w_j$, the vertical axis is the cost $f(w_j)$.

WHY. A function of one variable is just a curve, and "minimize" means "find the lowest point of the curve." That is something we can literally look at. This trick — freeze everything, wiggle one thing — is the heart of Coordinate Descent.

PICTURE. The plain prediction-error part alone is a smooth bowl (a parabola). Its lowest point sits at some value we'll call ${\rho }_j$ (defined properly in Step 3). The red dot marks that bottom.

Step 2 — The slope trick: how do you find the bottom?

WHAT. At the very bottom of a smooth curve, the curve is momentarily flat — its slope is zero. Rolling a marble: it stops where the ground is level.

WHY THIS TOOL (the slope / derivative). We want a lowest point. The one property that singles out a lowest point of a smooth bowl is "flat here." The derivative is exactly the machine that reports the slope at each spot. That is why we reach for it and not, say, plugging in random numbers: it turns "find the minimum" into "solve slope $=0$", an equation.

PICTURE. Three tangent lines: on the left the slope points down (negative), on the right it points up (positive), and at the bottom it is perfectly horizontal. Where the slope crosses from negative to positive is the minimum.

Step 3 — Define ${\rho }_j$: where the smooth part alone wants to sit.

WHAT. Take only the prediction-error bowl (ignore penalties for a second) and find its bottom. Standardize features so $\frac{1}{n}{\sum }_i{x}_{ij}^2=1$ (every feature has the same "size"). Then the bowl's slope is $\frac{d}{d{w}_j}[\frac{1}{2n}{\sum }_i(r_i-w_j{x}_{ij}{)}^2]=-{\rho }_j+w_j,{\rho }_j\equiv \frac{1}{n}{\sum }_i{x}_{ij}{r}_i.$

Read the symbols where they sit:

• ${\rho }_j$ = how strongly feature $j$ still correlates with the leftover $r_i$. Big ${\rho }_j$ → this feature explains a lot that no one else does.
• $-{\rho }_j+w_j$ = the slope. Set it to zero → $w_j={\rho }_j$. So without any penalty, the answer is just ${\rho }_j$.

WHY. ${\rho }_j$ is the single number that summarizes "what the data wants." Everything the penalties do from now on is push this number around. Naming it lets us watch that push.

PICTURE. The unpenalized bowl with its bottom pinned at $w_j={\rho }_j$ (red). This is our starting point before penalties bite.

Step 4 — Add the L2 penalty: the bowl gets narrower (a squeeze).

WHAT. Add $\frac{\lambda (1-\alpha )}{2}{w}_j^2$. This is another parabola centered at $0$. Adding two parabolas gives a parabola — but a steeper, narrower one whose bottom is pulled toward $0$.

Setting the combined smooth slope to zero: $-{\rho }_j+w_j+\lambda (1-\alpha )w_j=0⟹w_j=\frac{{\rho }_j}{1+\lambda (1-\alpha )}.$ Term by term:

• $-{\rho }_j+w_j$ = slope of the data bowl (Step 3).
• $+\lambda (1-\alpha )w_j$ = extra slope contributed by L2; it grows the farther you are from $0$, always pulling you back toward $0$.
• Result: divide ${\rho }_j$ by $1+\lambda (1-\alpha )>1$ — a proportional shrink. Nothing gets killed, everything just gets smaller.

WHY. L2 is smooth (a plain $w_j^2$), so it keeps the slope trick working. It is Ridge Regression's entire contribution: gentle, democratic shrinkage — never zero.

PICTURE. The original bowl (thin black) and the L2-tightened bowl (red), with the minimum sliding from ${\rho }_j$ toward $0$ but not reaching it.

Step 5 — The L1 penalty has a KINK, so the slope trick breaks. Introduce the subgradient.

WHAT. Now add $\lambda \alpha |w_j|$. The absolute value $|w_j|$ is a V shape with a sharp corner at $0$. On the right it has slope $+1$; on the left slope $-1$; at $0$ it has no single slope — the corner is ambiguous.

WHY A NEW TOOL (the subgradient). Our slope trick needs a slope, but the V has no slope at its tip. The subgradient repairs this: at the corner, instead of one slope we allow the whole range of slopes $[-1,+1]$ that could "lean against" the V there. Formally: $\partial |w_j|=\{\begin{array}{ll}+1& w_j>0\\ -1& w_j<0\\ [-1,+1]& w_j=0.\end{array}$ A minimum now means: zero is one of the allowed slopes ($0\in$ total subgradient), not that the slope equals zero.

PICTURE. The V of $|w_j|$ with the fan of dashed slope-lines pivoting through the corner — every line from slope $-1$ to $+1$ is "valid" there. That fan is exactly the set $[-1,1]$.

Step 6 — Solve with the corner: the soft-threshold appears.

WHAT. Write "zero is an allowed total slope" for the full cost: $0\in \underset{\text{smooth slope, from Steps 3–4}}{\underbrace{-{\rho }_j+w_j+\lambda (1-\alpha )w_j}}+\underset{\text{L1 corner term}}{\underbrace{\lambda \alpha \partial |w_j|}}.$

Now split into cases by the sign of $w_j$ — we must cover all of them:

• Case $w_j>0$: here $\partial |w_j|=+1$, so $0=-{\rho }_j+w_j(1+\lambda (1-\alpha ))+\lambda \alpha$, giving $w_j=\frac{{\rho }_j-\lambda \alpha }{1+\lambda (1-\alpha )}$. Valid only if the numerator ${\rho }_j-\lambda \alpha >0$, i.e. ${\rho }_j>\lambda \alpha$.
• Case $w_j<0$: here $\partial |w_j|=-1$, giving $w_j=\frac{{\rho }_j+\lambda \alpha }{1+\lambda (1-\alpha )}$. Valid only if ${\rho }_j<-\lambda \alpha$.
• Case $w_j=0$ (the dead zone): the corner allows any slope in $[-1,1]$. Zero is a minimum precisely when $-{\rho }_j+\lambda \alpha \cdot [-1,1]$ contains $0$, i.e. when $|{\rho }_j|\le \lambda \alpha$. The feature is killed.

Stitching the three cases together gives one clean formula — the soft-thresholding operator $S$: $\boxed{w_j=\frac{S({\rho }_j,\lambda \alpha )}{1+\lambda (1-\alpha )}},S(z,\gamma )=\mathrm{sign}(z)\max (|z|-\gamma ,0).$

Read it:

• $S({\rho }_j,\lambda \alpha )$ = "slide ${\rho }_j$ toward $0$ by $\lambda \alpha$; if you reach $0$, stop there." That is the L1 shrink-and-kill (numerator).
• $1+\lambda (1-\alpha )$ = the L2 proportional squeeze (denominator).

WHY. The three sign cases are forced by the V; folding them into $\max (|z|-\gamma ,0)$ is just bookkeeping. See Soft-Thresholding Operator for this shape in isolation.

PICTURE. The graph of $w_j$ versus ${\rho }_j$: a flat dead band of width $2\lambda \alpha$ around the origin (output stays $0$), then two straight lines of slope $\frac{1}{1+\lambda (1-\alpha )}$ shifted inward. The red segment is the dead zone.

Step 7 — Degenerate & limiting cases (so nothing surprises you).

WHAT / WHY / PICTURE for the corners of the knob-space:

• $\alpha =0$ (pure Ridge Regression): threshold $\lambda \alpha =0$, so $S({\rho }_j,0)={\rho }_j$. Update $={\rho }_j/(1+\lambda )$. Dead band has zero width → nothing is ever killed. The graph is a single tilted line through the origin.
• $\alpha =1$ (pure Lasso Regression): denominator $=1$. Update $=S({\rho }_j,\lambda )$. Pure shift, no squeeze; the dead band is at its widest for given $\lambda$.
• $\lambda =0$ (no penalty): both effects vanish, $w_j={\rho }_j$ — back to plain least squares (Step 3).
• $|{\rho }_j|=\lambda \alpha$ exactly (the boundary): numerator is exactly $0$, so $w_j=0$ — the feature sits right on the edge of the dead zone and gets dropped.

PICTURE. Three soft-threshold curves overlaid: Ridge ($\alpha =0$, no flat part), Elastic Net ($0<\alpha <1$, medium flat + shallow slope), Lasso ($\alpha =1$, wide flat + slope 1). The Elastic-Net middle curve is red.

The one-picture summary

One figure, the whole journey: start at ${\rho }_j$ (what data wants) → L1 slides it inward by $\lambda \alpha$ and may drop it into the dead zone → L2 divides what survives by $1+\lambda (1-\alpha )$. Two shrinks, one kill, in order.

Connections

• Elastic Net regularization — the parent this walkthrough derives.
• Soft-Thresholding Operator — the $S$ that Step 6 produces.
• Coordinate Descent — the "freeze all but one" strategy of Step 1.
• Ridge Regression — the $\alpha =0$ limit (Step 7).
• Lasso Regression — the $\alpha =1$ limit (Step 7).
• Feature Correlation & Multicollinearity — why the L2 squeeze matters.
• Bias-Variance Tradeoff — both shrinks trade variance for bias.
• Cross-Validation — how $\lambda ,\alpha$ get chosen in practice.

Reveal-practice: