Regularization — L1 (lasso), L2 (ridge), dropout: Visual walkthrough

This page rebuilds the single most important picture in regularization from nothing: why does the diamond-shaped penalty (L1) snap weights to exactly zero, while the round penalty (L2) only makes them smaller? We assume you have never seen a contour, a norm, or a gradient. We build each one, in order, on a picture.

Step 1 — Two knobs, and what "the loss" even is

WHAT. Our model has just two adjustable numbers, the weights $w_{1}$ and $w_{2}$ . Think of them as two dials. Every possible pair $(w_{1}, w_{2})$ is a single point on a flat sheet of paper — the horizontal axis is $w_{1}$ , the vertical axis is $w_{2}$ .

WHY. Before we can talk about penalties or shapes, we need a stage to draw them on. That stage is the plane of all possible weight choices. We use two weights (not one, not fifty) purely because a plane is the largest thing we can honestly draw.

PICTURE. Look at the figure: the sheet is the space of choices. One dot is marked — that is one particular model, say $(w_{1}, w_{2}) = (2, 1.5)$ .

Figure — Regularization — L1 (lasso), L2 (ridge), dropout

one point (w_{1}, w_{2}) = a fully-decided pair of dials one model

Step 2 — The data loss is a bowl; its shadow is a nest of ellipses

WHAT. For each choice $(w_{1}, w_{2})$ there is a number: how badly the model fits the training data. Call it $L_{data}$ . Small = good fit. If we plot that number upward out of the sheet, we get a bowl — low in one spot (the best fit) and rising as we move away.

WHY. The parent note wrote the data loss as an average of per-example errors,

L_{data} = \frac{1}{N} n = 1 \sum N ℓ (y_{n}, \overset{y}{^}_{n}) .

Let us define every symbol here, on this page:

$y_{n}$ ::: the true target of training example $n$ (e.g. the measured drag).
$\overset{y}{^}_{n}$ ::: the model's prediction for that example — for our linear model $\overset{y}{^}_{n} = w_{1} x_{1, n} + w_{2} x_{2, n}$ , so it depends on the dials.
$ℓ (y_{n}, \overset{y}{^}_{n})$ ::: the per-example loss, a number saying how far one prediction missed; we use the squared miss $ℓ = (y_{n} - \overset{y}{^}_{n})^{2}$ .
$\frac{1}{N} \sum_{n = 1}^{N}$ ::: average that miss over all $N$ training examples.

Because $\overset{y}{^}_{n}$ is linear in $w_{1}, w_{2}$ , a squared miss is a quadratic in the dials, and a quadratic in two variables is always a bowl (a paraboloid). We need this shape because it hands us the one tool that decodes the picture: contour lines.

PICTURE. We slice the bowl at equal heights and drop the slices flat onto the sheet. Each slice is an ellipse — a ring of "equally-bad" models. The tiny centre dot $\overset{w}{^}$ is the unconstrained best fit (bottom of the bowl).

L_{data} (w_{1}, w_{2}) = a fixed number (one chosen height) c ⟹ all models tied at badness c one ellipse

Here $c$ is just a constant level value we pick — "show me every dial-setting whose badness equals exactly $c$ ." Sweep $c$ upward and you get the whole nest of rings.

Why an ellipse and not a circle? Because the data usually pins down one combination of dials more tightly than another, so the bowl is steeper along one direction in weight space and flatter along another. A steeper direction means loss rises fast (a squashed ring); a flatter direction rises slowly (a stretched ring). A lopsided bowl gives lopsided rings — that is all "ellipse" means here.

Step 3 — "Dislike big weights" needs a ruler: the norm

WHAT. We now add a second wish: keep the weights small. To keep something small we must measure its size. A rule that turns a pair $(w_{1}, w_{2})$ into a single "size" number is called a norm.

WHY. We cannot penalize bigness without first defining bigness — and there is more than one honest way to measure it. That choice of ruler is exactly what separates L1 from L2. So we introduce two rulers side by side.

PICTURE. Two rulers, two shapes of "all points of size 1":

The L2 ruler (straight-line distance from the origin): $∥ w ∥_{2} = w_{1}^{2} + w_{2}^{2}$ . All points at distance $1$ form a circle.
The L1 ruler (add up the sideways + up moves, like walking city blocks): $∥ w ∥_{1} = ∣ w_{1} ∣ + ∣ w_{2} ∣$ . All points of size $1$ form a diamond.

Term-by-term: the L2 ruler is Pythagoras (a straight flight). The L1 ruler forbids diagonals — you must move along the axes, so a diagonal costs more. That extra cost on diagonals is precisely what pushes the diamond's corners onto the axes.

Step 4 — The constraint view: a fence around the origin

WHAT. There are two equivalent ways to say "keep weights small." One: add a penalty $λ \cdot ∥ w ∥$ to the loss (the parent note's view, where $λ$ = regularization strength). The equivalent, easier-to-draw way: forbid the weights from leaving a fenced region of size $C$ around the origin. Formally, minimise $L_{data}$ subject to $∥ w ∥ \leq C$ .

WHY. The penalty view lives up out of the page (it adds a bump to the bowl's height). The fence view lives flat on the page (a region you may not leave). Flat is drawable, so we pick it.

Why the two views are the same trade-off (intuitive sketch). Suppose the constrained answer lands on the fence (it does, whenever $\overset{w}{^}$ is outside). At that touching point the two views must agree on the balance between "fit better" and "stay small." Push the fence outward a little (raise $C$ ): you are allowed slightly bigger weights, so you can fit a little better — exactly what lowering the penalty $λ$ lets you do. Shrink the fence (lower $C$ ): you must accept a worse fit for smaller weights — exactly what raising $λ$ forces. So there is a monotone pairing:

PICTURE. Draw the fence for each ruler:

L2 fence = a filled disc of radius $C$ .
L1 fence = a filled diamond with corners at $(\pm C, 0)$ and $(0, \pm C)$ .

The best fit $\overset{w}{^}$ sits outside both fences (that is why we needed regularization). So we cannot sit at $\overset{w}{^}$ ; we must find the best allowed point touching the fence.

w min L_{data} (w) subject to stay inside the fence ∥ w ∥ \leq C

$min_{w}$ ::: search over all dial-settings.
$∥ w ∥ \leq C$ ::: the allowed region — inside or on the fence only.

Step 5 — Where an ellipse first kisses the fence

WHAT. Start the ellipses tiny at $\overset{w}{^}$ and let them grow outward. The first place a growing ellipse touches the fence is our answer: it is the lowest-loss point we are still allowed to occupy.

WHY. Growing the ellipse = accepting more data-error. We want the least extra error, so we stop the instant we reach the fence. Whichever point the ellipse touches first is the constrained solution — no picture-free algebra needed.

PICTURE (L2, the circle). The ellipse rolls out and first grazes the round fence at a point on its smooth edge. Nothing special happens there: both $w_{1}$ and $w_{2}$ are generally nonzero, just smaller than at $\overset{w}{^}$ . Round fence → smooth touch → shrunk, but nonzero.

Step 6 — The diamond kisses at a corner → an exact zero

WHAT. Repeat Step 5 with the diamond fence. Now the growing ellipse almost always meets the diamond at one of its sharp corners — and a corner of the L1 diamond lies exactly on an axis, where one weight is precisely $0$ .

WHY. A corner sticks out furthest toward the origin's incoming ellipses. An ellipse sweeping in from the outside reaches that protruding corner before it reaches the flat sides. This is a geometric fact about pointy shapes, not luck. And the corners of $∣ w_{1} ∣ + ∣ w_{2} ∣ \leq C$ are the four points $(\pm C, 0), (0, \pm C)$ — each has a coordinate equal to zero.

PICTURE. The ellipse touches the top corner $(0, C)$ : there $w_{1} = 0$ exactly. The model drops dial 1 completely — automatic feature selection, live, in one picture.

The same corner-snapping shows up algebraically as a soft-threshold rule when we optimise by gradient descent. We do not need it to see the zero — the picture already gave it — but it is worth naming, and it needs one new symbol:

Here is the rule, derived in the parent note's L1 section (we only quote and read it):

Contrast the L2 rule $w_{i}^{new} = (1 - η λ) w_{i}$ : that multiplies by a factor just under $1$ , so — like halving repeatedly — it scales toward zero but never lands on it. Round fence, round rule, no exact zeros.

Step 7 — The degenerate & edge cases (nothing left unshown)

We must cover the corners the reader could hit. Each gets its own panel in the figure.

Case A — $λ = 0$ (no fence). The fence has infinite size ( $C = ∥ \overset{w}{^} ∥$ suffices); the ellipse sits happily at $\overset{w}{^}$ . No shrinkage. Overfitting risk returns.

Case B — $λ \to \infty$ (fence shrinks to a dot). $C \to 0$ : the only allowed point is the origin. Both weights are zero — the model predicts a constant. Useless, as the parent warned.

Case C — the ellipse touches an L1 edge, not a corner. It can happen: if the loss ellipses are aligned just right, the first touch is on a flat side, giving two nonzero weights. So L1 does not guarantee zeros — it makes them overwhelmingly likely, because corners are reached first from almost every direction.

Case D — $w_{i} = 0$ exactly, where $∣ w_{i} ∣$ has a kink. The absolute-value ruler has no single slope at $0$ (a sharp "V"). We handle it with the subgradient: any slope between $- 1$ and $+ 1$ is allowed there. That undefined slope is exactly what lets a weight rest at zero instead of being pushed off it — a feature, not a bug.

The one-picture summary

The final figure lays the two fences side by side under the same incoming ellipses:

Left (L2, circle): smooth touch, both weights shrunk, neither zero → weight decay.
Right (L1, diamond): corner touch on the axis, one weight exactly zero → sparse selection.

Same loss, same $λ$ -sized fence — only the shape of the ruler changed, and that shape decided everything.

Recall Feynman retelling — say it to a 12-year-old

You have two dials and you want the machine to guess flight fuel well. Some dial settings guess well, some badly; draw that on paper as rings — inner rings are good guesses, outer rings bad. The best guess is a dot, but that dot uses huge, twitchy dials that break on new flights. So we draw a fence around the safe middle and say "you must stay inside." Now let the good-guess rings grow until one touches the fence — that touch is your answer. If the fence is a circle, the ring touches somewhere smooth and both dials come out small-but-on. If the fence is a diamond, the ring almost always hits a pointy corner, and those corners sit right on the lines where one dial is turned completely OFF. That is why the diamond ruler (L1) throws away useless dials, and the round ruler (L2) just turns everyone down a notch. Turn the fence tiny and both dials die; remove the fence and you're back to twitchy overfitting. Choosing the fence size is the only real decision.

Recall

What does the round (L2) fence do to the weights? ::: Shrinks both toward zero smoothly, but rarely to exactly zero (weight decay). Why does the diamond (L1) fence produce exact zeros? ::: Its corners lie on the axes, and a growing loss-ellipse touches a corner first from almost any direction, forcing a weight to be exactly 0. What is $\overset{w}{^}$ in these pictures? ::: The unconstrained best fit — the bottom of the data-loss bowl, centre of the ellipses. What does a larger $λ$ mean in the fence picture? ::: A smaller fence $C$ — a tighter constraint that shrinks weights more. What is $η$ in the soft-threshold rule? ::: The learning rate — the step size gradient descent takes each update. Give one case where L1 fails to zero anything. ::: When the ellipses touch a flat edge of the diamond instead of a corner, both weights stay nonzero (Case C).

Prerequisite & neighbour links: Gradient Descent Variants (the update rules whose geometry we drew), Feature Engineering (L1's automatic selection), Overfitting Detection (why we fence at all), Ensemble Methods and Neural Network Architectures (dropout as an ensemble of sub-networks), Bayesian Inference (penalties as priors), and pick $λ$ via Cross-Validation.