L2 (Ridge) regularization: Visual walkthrough

We assume nothing. If a symbol appears, we first say what it means and show what it looks like.

Step 0 — The picture we are trying to fit

Imagine you have some dots on a graph. Each dot is one example: a horizontal position $x$ (the thing you measured) and a vertical position $y$ (the thing you want to predict).

Look at the figure: the teal dots are data, the orange line is our guess $\overset{y}{^} = w x$ , and each plum stick is one residual. Fitting means: choose $w$ so the sticks are collectively as short as possible.

Step 1 — Measuring total wrongness: the sum of squared residuals

WHAT. We turn "the sticks are short" into a single number to minimize.

J_{OLS} (w) = i = 1 \sum n one stick, squared (y_{i} - w x_{i})^{2}

Reading it term by term:

$y_{i} - w x_{i}$ is one plum stick (a residual).
We square it so up-gaps and down-gaps both count as positive wrongness (a $- 3$ gap is just as bad as a $+ 3$ gap), and so big gaps hurt disproportionately.
$\sum_{i = 1}^{n}$ adds up the squared sticks over all $n$ dots.

WHY squares and not absolute value? Squaring makes $J$ a smooth parabola in $w$ — and parabolas have exactly one bottom you can find with calculus. Absolute value has a sharp corner (that road leads to L1 (Lasso) regularization, a different story).

PICTURE. Plot $J_{OLS}$ against the weight $w$ . It is a single bowl.

The teal bowl dips to its lowest point at the OLS weight $w_{OLS}$ — the slope that makes the sticks shortest.

Step 2 — Why the bowl can betray you: chasing noise

WHAT. Watch what happens when the data is a little noisy or the features nearly repeat each other.

WHY this matters. If two features carry almost the same information, the bowl's bottom becomes a long, nearly flat trench instead of a sharp point. A flat trench means: many wildly different weight pairs give almost the same fit. OLS is free to pick a crazy one — like $w_{1} = + 50, w_{2} = - 48$ — that happens to sit in the trench.

PICTURE. In 2D (two weights $w_{1}, w_{2}$ ) the cost is a set of nested contour ovals. Correlated features stretch those ovals into a long valley.

The orange ovals are lines of equal cost. Their long axis is the trench. The OLS minimum (orange dot) can drift far from the origin along that trench — huge, cancelling weights. That is the disease Ridge cures.

Step 3 — The fee: adding a penalty on weight size

WHAT. We charge a fee proportional to the squared length of the weight vector and add it to the cost.

Before the formula, two pieces of notation we are about to use:

J (w) = data fit (Step 1) i = 1 \sum n (y_{i} - w^{⊤} x_{i})^{2} + the fee λ j = 1 \sum d w_{j}^{2}

Term by term:

$w = (w_{1}, \dots, w_{d})$ is now a vector of weights (one per feature). $d$ is the number of features.
$w^{⊤} x_{i} = \sum_{k = 1}^{d} w_{k} x_{ik} = w_{1} x_{i 1} + \dots + w_{d} x_{i d}$ is the model's prediction for example $i$ : a dot-product that multiplies each feature $x_{ik}$ of that example by its weight $w_{k}$ and sums the results.
$\sum_{j} w_{j}^{2} = ∥ w ∥^{2}$ is the squared length of the weight vector — how far $w$ sits from the origin.
$λ \geq 0$ is the price per unit of squared length. Small $λ$ = cheap weights; huge $λ$ = weights are expensive.

WHY squared length? It is a smooth bowl centred at the origin — a second bowl we add on top of the data bowl. Small weights = near the origin = low fee. It grows fastest for the largest weights, so it punishes exactly the runaway $+ 50/ - 48$ pairs from Step 2.

PICTURE. The fee alone is its own bowl, lowest at $w = 0$ .

Step 4 — Two bowls add up, and the minimum slides home

WHAT. The Ridge cost is (data bowl) $+$ (fee bowl). Adding two upward bowls gives another upward bowl — but its bottom sits between the two.

WHY this is the whole point. The data bowl wants $w$ at $w_{OLS}$ (possibly far out in the trench). The fee bowl wants $w$ at the origin. Their sum compromises: the minimum is pulled from $w_{OLS}$ toward $0$ . Bigger $λ$ = steeper fee bowl = more pull. This is shrinkage.

PICTURE. In 1D, watch three bowls stack.

The teal data bowl (min at $w_{OLS}$ ), the plum fee bowl (min at $0$ ), and the orange total bowl whose min sits in between. The orange dot is the Ridge answer.

Step 5 — Do the calculus: find the bottom of the total bowl

WHAT. The bottom of a bowl is where its slope (gradient) is zero. We write the cost in matrix form and set the gradient to $0$ .

Matrix pieces (plain words):

$X$ = the data table, $n$ rows (examples) by $d$ columns (features).
$y$ = the column of $n$ targets.
$X w$ = all $n$ predictions at once.
$y - X w$ = all $n$ residual sticks at once.
$I$ = the == $d \times d$ identity matrix== — a $d$ -by- $d$ square with $1$ s on the diagonal and $0$ s elsewhere, the matrix version of the number $1$ (it leaves any vector unchanged). It is $d \times d$ so it lines up with $X^{⊤} X$ , which is also $d \times d$ ; only then can we add them.

J (w) = (y - X w)^{⊤} (y - X w) + λ w^{⊤} w

The two derivative rules we need, in plain words. The gradient $\nabla_{w}$ just collects the ordinary derivative with respect to each weight $w_{1}, \dots, w_{d}$ into a vector. Two little facts do all the work:

Rule A: $\frac{\partial}{\partial w} (w^{⊤} w) = 2 w$ . Reason: $w^{⊤} w = w_{1}^{2} + \dots + w_{d}^{2}$ , and the derivative of $w_{j}^{2}$ with respect to $w_{j}$ is $2 w_{j}$ — exactly like $\frac{d}{d w} w^{2} = 2 w$ in one variable. Stack those up and you get $2 w$ .
Rule B: $\frac{\partial}{\partial w} (w^{⊤} A w) = 2 A w$ when $A$ is symmetric. This is the vector version of $\frac{d}{d w} (a w^{2}) = 2 a w$ : the quadratic form $w^{⊤} A w$ is the multi-dimensional " $a w^{2}$ ," and its slope grows linearly as $2 A w$ .

Now expand and differentiate term by term. First multiply out the data-fit part:

(y - X w)^{⊤} (y - X w) = constant in w y^{⊤} y - linear in w 2 w^{⊤} X^{⊤} y + quadratic in w w^{⊤} (X^{⊤} X) w .

Differentiate each:

$y^{⊤} y$ has no $w$ in it → derivative $0$ .
$- 2 w^{⊤} X^{⊤} y$ is linear; the derivative of $w^{⊤} b$ is $b$ , so this gives $- 2 X^{⊤} y$ .
$w^{⊤} (X^{⊤} X) w$ is a quadratic form with the symmetric matrix $A = X^{⊤} X$ ; by Rule B its derivative is $2 X^{⊤} X w$ .
The fee $λ w^{⊤} w$ by Rule A gives $2 λ w$ .

Add them:

\nabla_{w} J = pull from data - 2 X^{⊤} y + data curvature 2 X^{⊤} X w + fee pull toward 0 2 λ w = 0

WHY the gradient? For a bowl, the single point where every direction is flat is the bottom — the global minimum. Because both bowls are convex, their sum has exactly one bottom, so this equation has one answer.

Cancel the $2$ 's and gather $w$ :

(X^{⊤} X + λ I) w = X^{⊤} y ⟹ \hat{w}_{ridge} = (X^{⊤} X + λ I)^{- 1} X^{⊤} y

PICTURE. The gradient is an arrow that points uphill; at the minimum the data's uphill push and the fee's inward push cancel.

At the orange dot the teal arrow (data wanting to move outward) and the plum arrow (fee wanting to move to origin) are equal and opposite — total gradient zero.

Step 6 — The ridge on the diagonal: why $+ λ I$ saves a broken problem

WHAT. Look at $X^{⊤} X + λ I$ . Adding $λ$ down the diagonal is where the name "ridge" comes from.

WHY it matters (degenerate case). If features are perfectly correlated, or if you have more features than examples ( $d > n$ ), then $X^{⊤} X$ is singular — you cannot invert it, OLS has no unique answer (the trench of Step 2 is perfectly flat, infinitely long). Adding $λ I$ with $λ > 0$ lifts the flat trench into a genuine bowl: $X^{⊤} X + λ I$ becomes positive definite → always invertible. This connects to Singular Value Decomposition: every singular value $σ_{j}^{2}$ becomes $σ_{j}^{2} + λ > 0$ .

PICTURE. A cross-section: the flat OLS trench versus the lifted Ridge bowl.

The grey flat line is the singular OLS trench (no unique bottom). Adding the plum fee tilts it into the orange bowl with one clear minimum.

Step 7 — Sweep $λ$ : all the edge cases at once

WHAT. Trace $\hat{w}$ as $λ$ goes from $0$ to $\infty$ .

$λ = 0$ : fee bowl vanishes → we recover Ordinary Least Squares. Full fit, maybe wild weights.
$0 < λ < \infty$ : weights shrink smoothly toward $0$ , never reaching exactly $0$ (that's L1 (Lasso) regularization, whose corners hit the axes).
$λ \to \infty$ : fee dominates, every weight $\to 0$ → flat, underfit model.

The sweet spot in the middle trades a little bias for a big drop in variance, and you find it with Cross-Validation — never with training error.

PICTURE. The shrinkage path: each weight's value drawn against $λ$ .

Every coloured curve is one weight. At the left ( $λ = 0$ ) they start at their OLS values (possibly large and opposite); as $λ$ grows they all slide continuously toward the zero line but never touch it.

The one-picture summary

One frame holds the whole derivation: the teal data contours pulling outward, the plum fee circle pulling inward, and the orange Ridge solution sitting at the tangency where the two forces balance — always closer to the origin than the OLS point, always unique.

Recall Feynman retelling — say it to a 12-year-old

You draw a line through some dots and measure how wrong you are by adding up the squared gaps — that's one bowl-shaped scorecard, lowest where the line fits best. Trouble: if two of your inputs are near-twins, that bowl flattens into a long ditch and the computer picks a crazy line, giant slopes that secretly cancel. So you add a second bowl that charges a fee for big slopes, lowest when the slopes are zero. Add the two bowls: the new lowest point is squeezed between "fit the data" and "stay small," so your slopes shrink to something calmer. To find that point you look for where the ground is flat in every direction — set the slope of the total bowl to zero — and out pops $(X^{⊤} X + λ I)^{- 1} X^{⊤} y$ . The little $λ I$ added down the diagonal is a magic lift: even when the plain problem has no unique answer, adding the ridge turns the flat ditch into a real bowl with one bottom. Crank $λ$ up and every slope glides toward zero but never quite lands there — smooth shrinking, not switching off.

Recall Quick self-check

What does $+ λ I$ do to a singular $X^{⊤} X$ ? ::: Lifts every eigenvalue by $λ > 0$ , making the matrix positive-definite and invertible → a unique solution Where does the Ridge minimum sit relative to OLS? ::: Between the OLS point and the origin — pulled toward $0$ , more so for larger $λ$ Why squared residuals and squared weights? ::: Both give smooth convex bowls, so their sum has exactly one global minimum found by setting the gradient to zero