Elastic Net regularization: Worked examples — AI-ML deep dive

Before we start, let us make sure every symbol is earned.

Look at the figure: the input $ρ_{j}$ comes in on the horizontal axis, the output $w_{j}$ leaves on the vertical. The flat red segment in the middle is the L1 dead zone; the slope of the sloped part (less than $1$ ) is the L2 squish. Every example below is just "where does my $ρ_{j}$ land on this graph?"

The scenario matrix

Every input to the update belongs to one of these cells. Our job is to hit them all.

Cell	What varies	Example
A. Positive, survives	$ρ_{j} > 0$ , $∣ ρ_{j} ∣ > λ α$	Ex 1
B. Negative, survives	$ρ_{j} < 0$ , $∣ ρ_{j} ∣ > λ α$ (sign flows through)	Ex 2
C. Killed (inside dead zone)	$∣ ρ_{j} ∣ \leq λ α$ → exactly $0$	Ex 3
D. Boundary / degenerate	$∣ ρ_{j} ∣ = λ α$ exactly, or $ρ_{j} = 0$	Ex 4
E. $α = 1$ limit (pure Lasso)	divisor $= 1$ , only threshold acts	Ex 5
F. $α = 0$ limit (pure Ridge)	no dead zone, only divisor acts	Ex 6
G. Correlated group (grouping effect)	two equal features share weight	Ex 7
H. Real-world word problem	build $ρ_{j}$ , apply update, interpret	Ex 8
I. Exam twist ( $λ \to \infty$ )	limiting behaviour of both knobs	Ex 9

Cell A — positive feature that survives

Cell B — negative feature (sign carries through)

Cell C — weak feature killed inside the dead zone

Cell D — the exact boundary and the zero input

Cell E — pure Lasso limit ( $α = 1$ )

Cell F — pure Ridge limit ( $α = 0$ )

Cell G — correlated group, the grouping effect

Worked example Ex 7 · two identical features

$x_{1} = x_{2}$ , OLS wants total weight $1$ Forecast: Lasso might give $(1, 0)$ . What does Elastic Net give, $(0.5, 0.5)$ or something lopsided?

Suppose after the shared shrink each of $w_{1}, w_{2}$ must satisfy $w_{1} + w_{2} = W$ for some total $W > 0$ , and we minimize the L2 part $w_{1}^{2} + w_{2}^{2}$ subject to that.

Set up the tie-break. Minimize $g = w_{1}^{2} + w_{2}^{2}$ with $w_{1} + w_{2} = W$ . Why this step? Along the correlated direction the data-fit and L1 terms are flat (see parent note), so only the strictly-convex L2 term decides.
Substitute $w_{2} = W - w_{1}$ : $g = w_{1}^{2} + (W - w_{1})^{2}$ . Why this step? Reduce to one variable.
Differentiate & set zero: $g^{'} (w_{1}) = 2 w_{1} - 2 (W - w_{1}) = 0 \Rightarrow w_{1} = W /2$ . Why this step? The minimum of a smooth convex 1-D function is where its derivative vanishes.
Result $w_{1} = w_{2} = W /2$ : for $W = 1$ , each gets $0.5$ . Why this step? Symmetry via convexity.

Verify: compare energies. Lasso split $(1, 0)$ : $g = 1^{2} + 0^{2} = 1$ . Even split $(0.5, 0.5)$ : $g = 0.25 + 0.25 = 0.5 < 1$ . So the L2 term strictly prefers the even split — that is the grouping effect made numerical. See Feature Correlation & Multicollinearity. ✓

The figure shows the flat trench along $w_{1} + w_{2} = 1$ (where data-fit + L1 are indifferent) and the L2 bowl whose lowest point on that trench sits dead-center at $(0.5, 0.5)$ .

Cell H — real-world word problem

Worked example Ex 8 · house-price model, feature "garage area"

You standardized all features. During Coordinate Descent you compute the partial residual correlation for garage area as $ρ_{gar} = 0.62$ . Your cross-validated knobs are $λ = 0.8, α = 0.75$ . Does the model keep the garage feature, and with what weight?

Forecast: guess "kept" or "dropped," and a ballpark weight.

Threshold $γ = λ α = 0.8 \cdot 0.75 = 0.6$ . Why this step? Survival toll from the L1 flavor.
Survival test $∣ 0.62 ∣ = 0.62 > 0.6$ → survives (barely). Why this step? Just clears the dead zone, so garage area is a genuinely useful predictor.
Numerator $S (0.62, 0.6) = 0.62 - 0.6 = 0.02$ . Why this step? L1 shrink.
Divisor $1 + λ (1 - α) = 1 + 0.8 \cdot 0.25 = 1.2$ . Why this step? L2 squish.
Result $w_{gar} = 0.02/1.2 = 0.01 \overline{6}$ . Why this step? Combine.

Verify: kept but with a tiny weight $\approx 0.017$ — consistent with a feature sitting right on the edge of usefulness. Interpretation: garage area helps a sliver; nudge $λ$ up a hair and it would be dropped. Units: everything is standardized, so $w$ is unitless "std-of- $y$ per std-of-feature." ✓

Cell I — exam twist: limiting behaviour as $λ \to \infty$

Worked example Ex 9 · what happens to

$w_{j}$ as $λ \to \infty$ ? Fix any $ρ_{j}$ and any $α \in (0, 1)$ . Take $λ$ huge.

Threshold $γ = λ α \to \infty$ . Why this step? The dead zone grows without bound, so eventually $∣ ρ_{j} ∣ < λ α$ for every feature.
Numerator $S (ρ_{j}, λ α) \to 0$ once $λ α > ∣ ρ_{j} ∣$ . Why this step? Everything falls inside the dead zone.
Result $w_{j} \to 0$ for all $j$ . Why this step? Numerator hits zero (divisor only makes it smaller).

Numeric checkpoint: with $ρ_{j} = 0.9, α = 0.5, λ = 100$ : $γ = 50 ≫ 0.9$ , so $S = 0$ and $w_{j} = 0$ .

Verify: the model collapses to the all-zero (intercept-only) predictor — maximum bias, minimum variance, the far end of the Bias-Variance Tradeoff. Twist answer: if instead $α = 0$ (pure Ridge), the numerator is $ρ_{j}$ (never killed) but the divisor $1 + λ \to \infty$ , so $w_{j} \to 0$ too — same destination, different road (shrink vs kill). ✓

Which knob should you actually pick?

Active recall

Recall Predict the cell before revealing

If $ρ_{j} = - 0.4$ and $λ α = 0.5$ , is $w_{j}$ zero? ::: Yes — $∣ - 0.4 ∣ = 0.4 \leq 0.5$ , inside the dead zone (Cell C/D), so $w_{j} = 0$ . With $α = 0$ , can any weight be exactly zero from the update alone? ::: No — threshold is $0$ , so soft-thresholding never kills; only a divisor shrinks (Cell F). Two identical features, total desired weight $2$ — Elastic Net split? ::: $(1, 1)$ — the L2 term minimizes $w_{1}^{2} + w_{2}^{2}$ at equality (Cell G). As $λ \to \infty$ with any fixed $α$ , every $w_{j} \to$ ? ::: $0$ — the model collapses to intercept-only (Cell I).

Connections

Elastic Net regularization — parent; this page enumerates its update's every case.
Soft-Thresholding Operator — the $S (z, γ)$ that draws the dead-zone graph.
Lasso Regression — the $α = 1$ cell (Ex 5).
Ridge Regression — the $α = 0$ cell (Ex 6).
Coordinate Descent — where $ρ_{j}$ comes from (Ex 8).
Feature Correlation & Multicollinearity — the grouping cell (Ex 7).
Bias-Variance Tradeoff — the limiting cell (Ex 9).
Cross-Validation — how the real knobs get chosen.