L1 (Lasso) regularization: Visual walkthrough

We derive the soft-thresholding rule $w^{⋆} = sign (z) max (∣ z ∣ - λ, 0)$ from absolute zero.

Step 1 — Draw the two costs we are balancing

WHAT. We have a single number $w$ we are allowed to choose (one feature's weight). Two things pull on it:

a data-fit cost that wants $w$ near a special number $z$ ,
a penalty cost that wants $w$ near $0$ .

WHY start with one weight? In the simplest setting (features that don't interfere — orthonormal features), every weight optimizes on its own. So mastering one $w$ is mastering the whole problem. That $z$ is exactly the value $w$ would take with no penalty — the plain least-squares answer.

PICTURE. Two chalk curves. The data-fit cost is a parabola (a valley) centred at $z$ : pull $w$ toward $z$ . The penalty cost is a V-shape centred at $0$ : pull $w$ toward $0$ .

Step 2 — Add the curves: the total cost has a KINK

WHAT. The thing we actually minimize is the sum of the parabola and the V. We add them pointwise to get one total-cost curve.

WHY does the shape matter? A smooth valley has a smooth bottom you find by "slope $= 0$ ". But the V has a sharp point at $w = 0$ . When we add a V to a parabola, that sharp point can survive and become the lowest point of the whole thing. A sharp lowest point sitting exactly at $0$ is the entire secret of Lasso.

PICTURE. Three curves stacked: the parabola (blue), the V (pink), and their sum (yellow). Notice the yellow sum still has a corner at $w = 0$ — the V's kink pokes through.

Step 3 — Slope hunting on the right side ( $w > 0$ )

WHAT. Assume for now $w > 0$ . Then $∣ w ∣ = w$ (absolute value of a positive number is itself), so the penalty is just $λ w$ , which is smooth. Take the slope and set it to zero.

WHY the slope tool? The derivative answers exactly one question: "at this $w$ , is the cost still going down?" The minimum of a smooth valley is where the cost stops falling — slope $= 0$ . We use it because on the region $w > 0$ there is no kink to worry about.

$\frac{dJ}{d w} = slope of parabola (w - z) + slope of λ w λ = 0 \Rightarrow w = z - λ$

$(w - z)$ ::: how far right of the valley bottom we are; pulls back left.
$+ λ$ ::: the penalty's constant rightward-pull that shifts the answer.

The catch: we assumed $w > 0$ . The answer $w = z - λ$ is only allowed if $z - λ > 0$ , i.e. $z > λ$ . If $z$ isn't that big, this solution is a lie and we discard it.

PICTURE. The parabola's bottom sits at $z$ ; adding the penalty slides the true minimum left by exactly $λ$ , landing at $z - λ$ — but only when that point is still positive.

Step 4 — Slope hunting on the left side ( $w < 0$ )

WHAT. Now assume $w < 0$ . Then $∣ w ∣ = - w$ (absolute value flips the sign of a negative), so the penalty is $- λ w$ and its slope is $- λ$ .

WHY redo it? Absolute value is a different function on each side of zero. Honesty demands we check both. This is the mirror image of Step 3.

$\frac{dJ}{d w} = (w - z) - λ = 0 \Rightarrow w = z + λ$

$- λ$ ::: now the penalty pulls rightward (toward zero from the left).

The catch: valid only if $w < 0$ , i.e. $z < - λ$ . If $z$ is not negative enough, discard.

PICTURE. Mirror of Step 3: valley bottom at negative $z$ , penalty slides the minimum right by $λ$ to $z + λ$ , valid only while it stays negative.

Step 5 — The trapped case: $- λ \leq z \leq λ$

WHAT. What if $z$ is small — between $- λ$ and $+ λ$ ? Then Step 3 wanted $z > λ$ (false) and Step 4 wanted $z < - λ$ (false). Neither smooth solution is allowed. The minimum has nowhere to go but the kink: $w = 0$ .

WHY does it get stuck at zero? At the kink the "slope" isn't a single number — it's a whole range (the subgradient): anything from $- λ$ to $+ λ$ . The minimum sits at the kink whenever the parabola's pull there, which is $- z$ , is small enough to be cancelled by some slope in that range. That condition is precisely $∣ z ∣ \leq λ$ .

PICTURE. For small $z$ , both one-sided walks (left and right of $0$ ) go uphill. The bottom of the yellow total-cost curve is the sharp corner sitting right on $w = 0$ .

Step 6 — Stitch the three cases into one formula

WHAT. Collect all three regions into a single rule.

Region	Winner	Result
$z > λ$	Step 3	$w = z - λ$
$z < - λ$	Step 4	$w = z + λ$
$- λ \leq z \leq λ$	Step 5	$w = 0$

WHY one formula? All three lines say the same thing: move $z$ toward $0$ by $λ$ , but never past $0$ . That's captured by:

$w^{⋆} = S_{λ} (z) = keep the direction sign (z) shrink λ, floor at 0 max (∣ z ∣ - λ, 0)$

$sign (z)$ ::: $+ 1$ if $z > 0$ , $- 1$ if $z < 0$ — remembers which side we came from.
$∣ z ∣ - λ$ ::: the shrink: pull the magnitude in by $λ$ .
$max (\cdot, 0)$ ::: the clamp: if shrinking would cross zero, stop at zero.

PICTURE. The soft-threshold graph: input $z$ (x-axis) vs output $w^{⋆}$ (y-axis). A flat dead-zone $[- λ, λ]$ pinned to $0$ , and two 45° lines offset by $λ$ outside it. Overlaid: Ridge's line through the origin that never flattens.

Step 7 — Why the geometry agrees: diamond corners

WHAT. The penalized problem is equivalent to "minimize the data-fit cost while keeping $∥ w ∥_{1} = ∣ w_{1} ∣ + ∣ w_{2} ∣ \leq t$ " for some budget $t$ . In 2-D that budget region is a diamond (rotated square). Ridge's budget $w_{1}^{2} + w_{2}^{2} \leq t$ is a circle.

WHY does shape decide sparsity? The data-fit cost has oval contours (level sets of constant error). The solution is where the smallest oval first kisses the budget region. A diamond has sharp corners sitting on the axes — and a corner on the $w_{1}$ -axis means $w_{2} = 0$ . Ovals reach out and touch a corner very easily. A circle is smooth: the touch point almost never lands exactly on an axis.

PICTURE. Left: blue diamond, pink ovals, first contact at a corner ⇒ $w_{2} = 0$ . Right: blue circle, ovals touch on a smooth edge ⇒ both weights nonzero.

The one-picture summary

WHAT. One figure compressing the whole story: the total-cost curve for three values of $z$ (big positive → shrunk, small → zeroed, negative → sign-kept), with arrows showing where each minimum lands, and the soft-threshold map beside it.

Recall Feynman retelling — the whole walkthrough in plain words

You have one dial, $w$ . A spring pulls it toward $z$ (the answer with no rules). A second, constant-strength rubber band pulls it toward $0$ . If $z$ is far from zero, the rubber band just drags the dial a fixed amount $λ$ closer to zero — that's shrinkage. But if $z$ is already close to zero (within $λ$ ), the rubber band wins completely and snaps the dial to exactly $0$ — the feature is deleted. Add a second dial and this "snap to zero" becomes the sharp corner of a diamond that the error-ovals love to land on. That corner-love is Lasso's whole magic: it doesn't just shrink weights, it deletes the small ones, leaving a short, honest model.

Recall Where do the three regions come from?

$∣ w ∣$ is a different smooth function on each side of $0$ , giving two "slope $= 0$ " answers ( $z - λ$ and $z + λ$ ), each valid only on its own side. When neither is valid — $∣ z ∣ \leq λ$ — the minimum is trapped at the non-differentiable kink $w = 0$ .

Compare with Elastic Net when features are correlated, and note the same corner mechanism works for Logistic Regression with an L1 penalty — the kink is in the penalty, not the loss.