L2 (Ridge) regularization: Worked examples — AI-ML deep dive

This page is a drill. The parent note built the theory; here we hit every kind of situation Ridge can throw at you — every sign of $λ$ , degenerate inputs, limiting values, correlated twins, a word problem, and an exam trap. Guess before you compute.

Before we start, a one-line reminder of the two formulas we lean on the whole way down.

The scenario matrix

Every Ridge problem is really one of these cells. Each worked example below is tagged with the cell(s) it covers.

#	Cell (case class)	What is special	Example
A	$λ = 0$	Ridge = plain OLS	Ex 1
B	$λ > 0$ , one feature	denominator inflated, weight shrinks	Ex 1
C	$λ \to \infty$	weight $\to 0$ (limiting behaviour)	Ex 2
D	Singular $X^{⊤} X$ ( $d > n$ or duplicate cols)	OLS undefined, Ridge still works	Ex 3
E	Correlated twins	huge cancelling weights → even split	Ex 4
F	Sign of correlation negative ( $\sum x_{i} y_{i} < 0$ )	shrinks a negative weight toward 0	Ex 5
G	Two independent features (diagonal $X^{⊤} X$ )	each weight shrinks by its own factor	Ex 6
H	SVD view / limiting factors	$σ^{2} / (σ^{2} + λ)$ per direction	Ex 7
I	Real-world word problem	choose $λ$ , standardize	Ex 8
J	Exam twist (penalized intercept trap)	why $w_{0}$ is excluded	Ex 9

Example 1 — Cells A & B: the scalar knob

Step 1. Use the scalar formula $w = \frac{\sum x _{i} y _{i}}{\sum x _{i}^{2} + λ} = \frac{10}{5 + λ}$ . Why this step? One feature means the whole matrix formula collapses to a single fraction — no inverse needed.

Step 2. $λ = 0 : w = \frac{10}{5} = 2$ (this is the OLS answer — Cell A). Why this step? Setting $λ = 0$ removes the penalty entirely, so we must land back on plain least squares.

Step 3. $λ = 5 : w = \frac{10}{10} = 1$ . The weight halved (Cell B). Why this step? Adding $λ$ to the denominator is the entire mechanism of shrinkage — bigger denominator, smaller weight.

Step 4. $λ = 15 : w = \frac{10}{20} = 0.5$ . Why this step? Shows shrinkage is smooth and continuous, not a cliff.

Recall Verify

The scalar Ridge loss is $J (w) = \sum_{i} (y_{i} - w x_{i})^{2} + λ w^{2}$ . Differentiate term by term: $\frac{d}{d w} \sum_{i} (y_{i} - w x_{i})^{2} = \sum_{i} 2 (y_{i} - w x_{i}) (- x_{i}) = - 2 \sum_{i} x_{i} y_{i} + 2 w \sum_{i} x_{i}^{2}$ , and $\frac{d}{d w} λ w^{2} = 2 λ w$ . Adding them and setting to zero: $- 2 \sum_{i} x_{i} y_{i} + 2 w (\sum_{i} x_{i}^{2} + λ) = 0$ . Plug our numbers ( $\sum x_{i} y_{i} = 10, \sum x_{i}^{2} = 5, λ = 5$ ): $- 20 + 2 w (10) = 0 \Rightarrow 20 w = 20 \Rightarrow w = 1$ . ✓ Matches Step 3.

Example 2 — Cell C: the $λ \to \infty$ limit

Step 1. $λ \to \infty lim \frac{10}{5 + λ} = 0$ . Why this step? Numerator fixed, denominator $\to \infty$ , so the ratio squeezes to $0$ — this is the maximum-bias / underfit extreme.

Step 2. No finite $λ$ gives exactly $0$ : solving $\frac{10}{5 + λ} = 0$ needs $10 = 0$ , impossible. Why this step? Confirms the "shrinks toward but never reaches zero" claim — Ridge asymptotes, it doesn't snap.

Step 3. Solve $\frac{10}{5 + λ} = 0.1 \Rightarrow 5 + λ = 100 \Rightarrow λ = 95$ . Why this step? Concrete demo that reaching near-zero needs a large $λ$ .

Recall Verify

$\frac{10}{5 + 95} = \frac{10}{100} = 0.1$ . ✓ And units: $λ$ shares units of $\sum x_{i}^{2}$ (a "variance budget"), consistent.

Example 3 — Cell D: singular $X^{⊤} X$ , OLS dies, Ridge lives

Step 1. Build $X^{⊤} X$ . With $X = [11]$ , $X^{⊤} X = [1111]$ . Its determinant is $1 \cdot 1 - 1 \cdot 1 = 0$ → singular → OLS's $(X^{⊤} X)^{- 1}$ does not exist. Why this step? Shows exactly why plain OLS fails: you cannot invert a zero-determinant matrix.

Step 2. Add $λ I$ : $X^{⊤} X + I = [2112]$ . Determinant $= 2 \cdot 2 - 1 \cdot 1 = 3 \neq = 0$ → invertible. Why this step? This is the "ridge on the diagonal" cure — $λ$ lifts the diagonal so the matrix becomes positive-definite. See Singular Value Decomposition for the deep reason.

Step 3. Inverse: $[2112]^{- 1} = \frac{1}{3} [2 - 1 - 1 2]$ . And $X^{⊤} y = [11] \cdot 2 = [22]$ . Why this step? $2 \times 2$ inverse is $\frac{1}{d e t} [d - c - b a]$ — plug in and go.

Step 4. $\hat{w} = \frac{1}{3} [2 - 1 - 1 2] [22] = \frac{1}{3} [22] = [2/3 2/3]$ . Why this step? Ridge returned a unique, symmetric answer where OLS had none.

Recall Verify

Symmetry: identical columns must get identical weights, so $w_{1} = w_{2}$ . ✓ Fit: prediction $= 1 \cdot \frac{2}{3} + 1 \cdot \frac{2}{3} = \frac{4}{3} \approx 1.33$ , pulled below $y = 2$ because $λ > 0$ trades a little fit for smaller weights. ✓

Example 4 — Cell E: correlated twins, cancelling weights tamed

Step 1. Both OLS candidates fit training data: $50 \cdot x - 48 \cdot x = 2 x$ and $1 \cdot x + 1 \cdot x = 2 x$ . Identical predictions. Why this step? Establishes that fit alone cannot distinguish them — the penalty must decide.

Step 2. Compare penalties: $5 0^{2} + 4 8^{2} = 2500 + 2304 = 4804$ versus $1^{2} + 1^{2} = 2$ . Why this step? Ridge adds $λ \sum w_{j}^{2}$ to the cost; the cancelling pair is $2402 \times$ more expensive, so Ridge will run from it.

Step 3. Now compute the actual Ridge weights. Build $X = [1212]$ . Then $X^{⊤} X = [1122] [1212] = [5555]$ and $X^{⊤} y = [1122] [24] = [1010]$ . Why this step? The column dot products fill $X^{⊤} X$ ; note it's singular (det $= 25 - 25 = 0$ ), so plain OLS again has no unique answer — Ridge must rescue it.

Step 4. Add $λ I = I$ : $X^{⊤} X + I = [6556]$ , det $= 36 - 25 = 11$ , inverse $= \frac{1}{11} [6 - 5 - 5 6]$ . Why this step? $λ = 1$ lifts the diagonal from $5$ to $6$ , making the matrix invertible.

Step 5. $\hat{w} = \frac{1}{11} [6 - 5 - 5 6] [1010] = \frac{1}{11} [1010] = [10/11 10/11] \approx (0.909, 0.909)$ . Why this step? This is the concrete answer. It is the even split $(1, 1)$ pulled slightly toward zero — the "tiny shrink" is exactly the factor $\frac{10}{11}$ , i.e. each weight is $1$ scaled by $\frac{signal}{signal + λ}$ in disguise.

Recall Verify

Symmetry forces $w_{1} = w_{2}$ . Solve the reduced 1-feature Ridge: $w = \frac{10}{10 + 1} = \frac{10}{11} \approx 0.909$ . ✓ And $\frac{10}{11} < 1$ , so it's the even split shrunk toward zero, exactly as claimed. ✓

Example 5 — Cell F: shrinking a negative weight

Step 1. $λ = 0 : w = \frac{- 8}{4} = - 2$ . Why this step? Negative numerator → negative OLS weight; the feature pushes $y$ down.

Step 2. $λ = 4 : w = \frac{- 8}{4 + 4} = \frac{- 8}{8} = - 1$ . Why this step? Denominator inflates exactly as before — magnitude halves, sign unchanged.

Step 3. So $w : - 2 \to - 1 \to 0^{-}$ as $λ$ grows. Why this step? Shrinkage means "toward zero," which for a negative weight means moving up toward $0$ , not toward $+ \infty$ .

Recall Verify

$\frac{- 8}{8} = - 1$ ✓; sign preserved ( $- 2$ and $- 1$ both negative) ✓; $∣ - 1∣ < ∣ - 2∣$ so magnitude shrank ✓.

Example 6 — Cell G: two independent features shrink separately

Step 1. $X^{⊤} X + λ I = [5002]$ — still diagonal, so it decouples. Why this step? A diagonal matrix inverts entrywise; the two features don't interfere.

Step 2. $w_{1} = \frac{8}{4 + 1} = \frac{8}{5} = 1.6$ ; $w_{2} = \frac{3}{1 + 1} = \frac{3}{2} = 1.5$ . Why this step? Each feature follows its own scalar rule $w_{j} = \frac{( X ^{⊤} y ) _{j}}{( X ^{⊤} X ) _{j j} + λ}$ .

Step 3. Compare to OLS: $w_{1}^{O L S} = 8/4 = 2$ (dropped to 1.6, a $20%$ cut); $w_{2}^{O L S} = 3/1 = 3$ (dropped to 1.5, a $50%$ cut). Why this step? Reveals the key law: features with a small diagonal entry shrink harder. The same $λ = 1$ is a bigger fraction of a small diagonal entry.

Recall Verify

$8/5 = 1.6$ ✓, $3/2 = 1.5$ ✓. Fractional shrink: strong $1.6/2 = 0.8$ , weak $1.5/3 = 0.5$ → weak shrinks more (0.5 < 0.8) ✓.

Example 7 — Cell H: SVD shrinkage factors, both limits

First, the one definition this example needs.

Step 1. Strong: $\frac{σ _{1}^{2}}{σ _{1}^{2} + λ} = \frac{100}{100 + 1} = \frac{100}{101} \approx 0.990$ . Why this step? Large $σ^{2}$ dwarfs $λ$ , so the ratio $\approx 1$ → this direction is kept.

Step 2. Noisy: $\frac{σ _{2}^{2}}{σ _{2}^{2} + λ} = \frac{0.25}{0.25 + 1} = \frac{0.25}{1.25} = 0.2$ . Why this step? Small $σ^{2}$ is swamped by $λ$ , so the ratio $≪ 1$ → this direction is shrunk away.

Step 3. Limits: as $λ \to 0$ , both factors $\to 1$ (recover OLS, Cell A). As $λ \to \infty$ , both $\to 0$ (Cell C). Why this step? Confirms the factor smoothly interpolates the two extremes we already met.

Recall Verify

$100/101 \approx 0.9901$ ✓; $0.25/1.25 = 0.2$ ✓; both $\to 1$ at $λ = 0$ and $\to 0$ at $λ = \infty$ ✓.

Example 8 — Cell I: real-world word problem

Step 1. Say the true effect of area needs weight $w_{area}$ so that $w_{area} \times (area \sim 100)$ matters. To move price the same amount, $w_{area}$ can be $\sim 100 \times$ smaller than $w_{rooms}$ (which multiplies $\sim 3$ ). Why this step? Weight magnitude is inversely tied to feature scale — big-valued features earn tiny weights.

Step 2. The penalty $λ (w_{area}^{2} + w_{rooms}^{2})$ then barely touches the tiny $w_{area}$ but hammers $w_{rooms}$ . If $w_{rooms} = 6$ and $w_{area} = 0.06$ , penalty contributions are $6^{2} = 36$ vs $0.0 6^{2} = 0.0036$ — a $10000 \times$ imbalance. Why this step? Shows the penalty is scale-unfair: it silently favours large-valued features by leaving their (tiny) weights almost unpenalized.

Step 3. Fix: standardize each feature to zero mean, unit variance. Now both features live on the same scale and $λ$ penalizes them equally. Choose $λ$ by Cross-Validation, never by training error. Why this step? Standardization makes $\sum w_{j}^{2}$ compare apples to apples; CV picks the valley of the validation curve.

Recall Verify

Penalty ratio raw $= 6^{2} /0.0 6^{2} = 10 0^{2} = 10000$ ✓ — precisely the square of the scale ratio $100$ , confirming the unfairness scales with (unit ratio) $^{2}$ .

Example 9 — Cell J: exam twist, penalizing the intercept

Step 1. No penalty: minimize $\sum_{i} (y_{i} - w_{0})^{2}$ . Differentiate: $\frac{d}{d w _{0}} \sum_{i} (y_{i} - w_{0})^{2} = - 2 \sum_{i} (y_{i} - w_{0}) = 0 \Rightarrow \sum_{i} y_{i} = n w_{0} \Rightarrow w_{0} = \overset{y}{ˉ} = 5$ . Correct baseline. Why this step? The unpenalized intercept is just the mean of $y$ — exactly what a baseline should be.

Step 2. Penalized: minimize $\sum_{i} (y_{i} - w_{0})^{2} + λ w_{0}^{2}$ . Since every $x_{0} = 1$ , this is the scalar formula with " $\sum x_{0}^{2}$ " $= \sum_{i} 1 = n$ : $w_{0} = \frac{\sum _{i} y _{i}}{n + λ} = \frac{n y ˉ}{n + λ} = \frac{4 \cdot 5}{4 + 4} = \frac{20}{8} = 2.5$ . Why this step? Penalizing $w_{0}$ drags the baseline toward $0$ — here from $5$ down to $2.5$ , a systematic error for no benefit.

Step 3. Conclusion: exclude $w_{0}$ from the penalty (or center $y$ first). The intercept sets the output level, not the model's complexity. Why this step? Complexity lives in the slopes; shrinking the level just biases every prediction toward zero.

Recall Verify

Unpenalized $w_{0} = \overset{y}{ˉ} = 5$ ✓; penalized $= 20/8 = 2.5$ ✓; $2.5 < 5$ so it's biased toward zero ✓.

Recall

Recall Quick self-test

Scalar Ridge weight with $\sum x_{i} y_{i} = 10, \sum x_{i}^{2} = 5, λ = 5$ ::: $10/10 = 1$ Why does OLS fail on one example with two features? ::: $X^{⊤} X$ is singular (det $= 0$ ), not invertible Ridge answer for $x = (1, 1), y = 2, λ = 1$ ::: $(2/3, 2/3)$ Exact Ridge weights for the twin features $(1, 1) : 2, (2, 2) : 4$ at $λ = 1$ ::: $(10/11, 10/11) \approx (0.909, 0.909)$ Which shrinks more: strong ( $σ^{2} = 4$ ) or weak ( $σ^{2} = 1$ ) feature at $λ = 1$ ? ::: the weak one (factor $1/2$ vs $4/5$ ) SVD shrinkage factor at $σ = 10, λ = 1$ ::: $100/101 \approx 0.990$ Penalized intercept for $\overset{y}{ˉ} = 5, n = 4, λ = 4$ ::: $20/8 = 2.5$ (biased below the mean — don't do this) Does any finite $λ$ set a scalar Ridge weight exactly to $0$ ? ::: No — it only asymptotes to $0$