L1 (Lasso) regularization: Worked examples — AI-ML deep dive

This page hammers the soft-thresholding operator from the parent note $S_{λ} (z) = sign (z) max (∣ z ∣ - λ, 0)$ against every kind of input it can receive. Read the parent L1 (Lasso) regularization first if the symbols $z$ (the OLS coefficient before penalty), $λ$ (penalty strength), and $∥ w ∥_{1}$ are new to you — here we assume you know them and we grind through cases.

The scenario matrix

Every cell below is a distinct behaviour of Lasso. The worked examples that follow are tagged with the cell they hit.

#	Cell (case class)	Trigger condition	Expected behaviour
C1	Positive, large	$z > λ > 0$	shrink down: $w = z - λ > 0$
C2	Negative, large	$z < - λ$	shrink up: $w = z + λ < 0$
C3	Inside the band	$- λ \leq z \leq λ$	clamp to exactly 0
C4	Boundary knife-edge	$	z
C5	Zero input	$z = 0$	stays 0 for any $λ$
C6	No penalty	$λ = 0$	recovers OLS: $w = z$
C7	Infinite penalty	$λ \to \infty$	all weights $\to 0$ (predict mean)
C8	Vector / feature selection	many $z_{j}$ , one $λ$	some survive, some deleted
C9	Scaling trap (word problem)	features in different units	wrong features dropped unless standardized
C10	Exam twist	Ridge vs Lasso same $z$	Ridge never hits 0; Lasso does

We now cover all ten cells.

The picture that governs everything

Worked examples

C1 — Positive, large: plain shrinkage

C2 — Negative, large: sign is preserved

C3 — Inside the band: feature deleted

C4 — Knife-edge: exactly $∣ z ∣ = λ$

C5 — Zero input stays zero

C6 — $λ = 0$ : Lasso becomes OLS

C7 — $λ \to \infty$ : everything dies

C8 — The full vector: automatic feature selection

C9 — Word problem: the scaling trap

Worked example Housing price:

area and distance in different units A model predicts house price. Feature A = floor area, feature B = distance to station. Suppose the true effect of both is identical in real-world terms, but you measure area in square-metres and distance in metres. After OLS you get $z_{area} = 2.0$ and $z_{dist} = 0.3$ purely because the raw distance numbers are large (so their coefficient is small). Use $λ = 0.5$ .

Forecast: Both features genuinely matter equally — will Lasso keep both?

Unscaled: $w_{area} = S_{0.5} (2.0) = 1.5$ (survives), $w_{dist} = S_{0.5} (0.3) = 0$ (dropped!). Why this step? The L1 tax $λ ∣ w_{j} ∣$ is the same size for every coefficient, so a feature whose units force a tiny coefficient looks "cheap to drop" even if it's important.
Fix — standardize both features to unit variance. Then equal true effect ⇒ equal $∣ z_{j} ∣$ ; say both become $z = 1.2$ . Now $w = S_{0.5} (1.2) = 0.7$ for both. Why this step? Standardizing removes the unit artefact so the penalty compares like with like.

Answer: Unscaled drops distance ( $0$ ); scaled keeps both at $0.7$ . Verify: The figure shows the two dead-zone bands are identical width $λ$ , but the unscaled distance-coefficient $0.3$ falls inside it while $2.0$ escapes — a pure units artefact. Always standardize before Lasso. ✓

C10 — Exam twist: Ridge vs Lasso on the same small $z$

Active recall

Recall Which matrix cell does

$z = 0.5, λ = 0.5$ land in, and what is $w$ ? C4 (knife-edge). $max (0, 0) = 0$ , so $w = 0$ . The band is inclusive.

Recall Why does the scaling trap (C9) drop an important feature?

The L1 tax $λ ∣ w_{j} ∣$ is identical for every coefficient, so a feature whose units force a small coefficient looks cheap to zero out. Standardize first.

Recall On the same

$z = 0.4$ , why does Ridge give $0.2$ but Lasso gives $0$ ? Ridge shrinks proportionally ( $w = z / (1 + 2 λ)$ , smooth at 0); Lasso has a dead zone (kink) that clamps small $z$ exactly to 0.