L2 (Ridge) regularization: Exercises — AI-ML deep dive

This is the practice set for Ridge regression. Every problem is stated cleanly, then fully solved inside a collapsible Solution callout so you can test yourself first. Problems climb from L1 Recognition (do you know the pieces?) up to L5 Mastery (can you invent and combine?). After each level there is one [!mistake] callout naming the trap most people fall into there.

Everything here rests on two facts from the parent note, restated so nothing is assumed:

Level 1 — Recognition

L1.1

Write the Ridge cost for a single feature $x$ (no intercept), with data pairs $(x_{i}, y_{i})$ for the $n$ training examples and strength $λ$ . Then identify which term is the "data fit" and which is the "penalty".

Recall Solution

With one weight $w$ and one feature, the prediction for example $i$ is $w x_{i}$ . The squared error over all $n$ points (recall $n$ = number of training examples) plus the penalty is $J (w) = data fit i = 1 \sum n (y_{i} - w x_{i})^{2} + penalty λ w^{2} .$ The first sum measures how far predictions land from truth; the second charges a fee $λ$ for every unit of $w^{2}$ used.

L1.2

For $λ = 0$ , what does Ridge reduce to? For $λ \to \infty$ , what does $\hat{w}$ approach?

Recall Solution

$λ = 0$ : the penalty term vanishes, $J (w) = ∥ y - X w ∥^{2}$ — this is exactly Ordinary Least Squares.
$λ \to \infty$ : the fee on weight-size dominates everything, so the cheapest choice is $w \to 0$ . Every weight is squeezed toward zero (maximum underfit).

L1.3

True or false, with a one-line reason each: (a) Ridge can set a weight exactly to $0$ . (b) Adding $λ I$ can make a non-invertible $X^{⊤} X$ invertible.

Recall Solution

(a) False. Ridge's penalty is a smooth circle/sphere; it shrinks weights continuously but generically never hits exactly $0$ . Exact zeros are Lasso's specialty (its diamond has corners on the axes). (b) True. $X^{⊤} X$ is positive semi-definite (eigenvalues $\geq 0$ ). Adding $λ I$ with $λ > 0$ lifts every eigenvalue by $λ$ , making them all strictly positive → the matrix is invertible.

Level 2 — Application

L2.1

Scalar Ridge. You are given $\sum x_{i} y_{i} = 12$ and $\sum x_{i}^{2} = 4$ . Compute $w$ for $λ = 0$ , $λ = 2$ , and $λ = 8$ .

Recall Solution

From L1.1, setting $\frac{dJ}{d w} = 0$ gives $- 2 \sum x_{i} y_{i} + 2 w \sum x_{i}^{2} + 2 λ w = 0$ , so $w = \frac{\sum x _{i} y _{i}}{\sum x _{i}^{2} + λ} .$

$λ = 0$ : $w = 12/4 = 3$ (this is OLS).
$λ = 2$ : $w = 12/ (4 + 2) = 12/6 = 2$ .
$λ = 8$ : $w = 12/ (4 + 8) = 12/12 = 1$ . Notice the pattern: Ridge only inflates the denominator. Bigger $λ$ → bigger denominator → smaller $∣ w ∣$ , monotonically toward $0$ .

L2.2

Two-feature closed form. Let $X = 101011, y = 202, λ = 1.$ Compute $\hat{w} = (X^{⊤} X + λ I)^{- 1} X^{⊤} y$ .

Recall Solution

Step 1 — build $X^{⊤} X$ . Multiply columns of $X$ pairwise: $X^{⊤} X = [1 + 0 + 1 0 + 0 + 1 0 + 0 + 1 0 + 1 + 1] = [2112] .$ Step 2 — add $λ I$ (here $λ = 1$ , so $λ I$ adds $λ = 1$ to each diagonal entry): $X^{⊤} X + λ I = [3113] .$ Step 3 — build $X^{⊤} y$ . Dot each column of $X$ with $y$ : $X^{⊤} y = [1 (2) + 0 (0) + 1 (2) 0 (2) + 1 (0) + 1 (2)] = [42] .$ Step 4 — invert and multiply. For a $2 \times 2$ matrix $[a c b d]$ the inverse is $\frac{1}{a d - b c} [d - c - b a]$ . Here $a d - b c = 9 - 1 = 8$ : $(X^{⊤} X + λ I)^{- 1} = \frac{1}{8} [3 - 1 - 1 3] .$ $\hat{w} = \frac{1}{8} [3 - 1 - 1 3] [42] = \frac{1}{8} [12 - 2 - 4 + 6] = \frac{1}{8} [102] = [1.25 0.25] .$

L2.3

Feature-scaling fairness. Feature $x_{1}$ is a length in meters; you rescale it to millimeters (multiply by $1000$ ). To keep the same predictions, the weight $w_{1}$ must be divided by $1000$ . If the penalty is $λ w_{1}^{2}$ , by what factor does the penalty on this feature change?

Recall Solution

New weight $w_{1}^{'} = w_{1} /1000$ . Penalty term $λ (w_{1}^{'})^{2} = λ w_{1}^{2} /1 0^{6}$ . So the penalty on this one feature is cut by a factor of $1 0^{6}$ even though nothing about the underlying relationship changed. That is exactly why we standardize first — otherwise the penalty is a lottery decided by measurement units. See Feature Scaling / Standardization.

Level 3 — Analysis

L3.1

Prove monotone shrinkage. Show that in the scalar case $w (λ) = \frac{c}{s + λ}$ (with $c = \sum x_{i} y_{i}$ , $s = \sum x_{i}^{2} > 0$ ) has $∣ w ∣$ strictly decreasing in $λ$ for $λ \geq 0$ when $c \neq = 0$ , handle the edge case $c = 0$ , and find $lim_{λ \to \infty} w$ .

Recall Solution

Edge case $c = 0$ first. If $c = \sum x_{i} y_{i} = 0$ , then $w (λ) = 0/ (s + λ) = 0$ for every $λ \geq 0$ . So $∣ w ∣ \equiv 0$ is constant, not strictly decreasing — there is simply nothing to shrink. Keep this exception in mind; the strict-decrease claim needs $c \neq = 0$ . Main case $c \neq = 0$ . We want the sign of $\frac{d ∣ w ∣}{d λ}$ , so write the magnitude directly. Since $s + λ > 0$ , $∣ w (λ) ∣ = \frac{∣ c ∣}{s + λ} .$ This is a genuine function of $λ$ (no absolute-value kink, because the denominator never changes sign). Differentiate it: $\frac{d ∣ w ∣}{d λ} = ∣ c ∣ \cdot \frac{d}{d λ} (s + λ)^{- 1} = - \frac{∣ c ∣}{( s + λ ) ^{2}} .$ With $c \neq = 0$ we have $∣ c ∣ > 0$ and $(s + λ)^{2} > 0$ , so $\frac{d ∣ w ∣}{d λ} < 0$ for all $λ \geq 0$ . A function with strictly negative derivative is strictly decreasing — exactly what we set out to prove. (This is cleaner than tracking $d w / d λ$ and signs of $c$ separately: differentiating $∣ w ∣$ itself leaves no gap.) Limit. As $λ \to \infty$ the denominator $s + λ \to \infty$ , so $w (λ) = c / (s + λ) \to 0$ . More penalty → smaller, calmer weight, moving toward zero and never overshooting past it.

L3.2

Correlated twins. Suppose two features are identical: $x_{2} = x_{1}$ , and any weights with $w_{1} + w_{2} = 2$ fit the training data equally well (they all produce the same predictions). Among all such pairs, which one does Ridge choose, and what is its penalty?

Recall Solution

Ridge breaks the tie by the smallest penalty among equally-good fits. We minimize $w_{1}^{2} + w_{2}^{2}$ subject to $w_{1} + w_{2} = 2$ . Substitute $w_{2} = 2 - w_{1}$ : $P (w_{1}) = w_{1}^{2} + (2 - w_{1})^{2} = 2 w_{1}^{2} - 4 w_{1} + 4.$ Set $P^{'} (w_{1}) = 4 w_{1} - 4 = 0 \Rightarrow w_{1} = 1$ , hence $w_{2} = 1$ . Penalty $= 1^{2} + 1^{2} = 2$ . Compare a cancelling solution like $w_{1} = 50, w_{2} = - 48$ (also on some fitting line for near-twins): penalty $= 5 0^{2} + 4 8^{2} = 4804$ . Ridge overwhelmingly prefers the even split $w = (1, 1)$ . Read the figure. The solid black line is the set of all fitting pairs ( $w_{1} + w_{2} = 2$ ). The dashed circle is the smallest weight-budget circle that still touches this line; it kisses the line at exactly the point nearest the origin. The red dot marks that tangency point $(1, 1)$ — the even split. Every other fitting pair (like the far black dot toward $(2.6, - 0.6)$ ) lies on a bigger circle → larger penalty. This is why Ridge spreads weight across correlated features instead of letting them explode in opposite directions.

L3.3

SVD shrinkage direction. Using the parent's result, the effective factor along singular direction $j$ is $f_{j} = \frac{σ _{j}^{2}}{σ _{j}^{2} + λ}$ . For $λ = 4$ , compute $f_{j}$ for a strong direction $σ = 10$ and a weak direction $σ = 1$ . Which is shrunk more?

Recall Solution

Strong: $f = \frac{1 0 ^{2}}{1 0 ^{2} + 4} = \frac{100}{104} \approx 0.9615$ — kept almost intact.
Weak: $f = \frac{1 ^{2}}{1 ^{2} + 4} = \frac{1}{5} = 0.2$ — crushed to one-fifth. The weak (small- $σ$ ) direction is shrunk far more. Small $σ$ means that direction was measured with little energy in the data — it is where noise dominates. Ridge distrusts it and keeps the confidently-measured strong direction. Read the figure. The black curve plots the kept fraction $f = σ^{2} / (σ^{2} + λ)$ against the singular value $σ$ (with $λ = 4$ ). It rises from $0$ (at $σ = 0$ , fully crushed) toward $1$ (the dotted line, kept in full) as $σ$ grows. The two red dots mark our cases: the weak $σ = 1$ sits low at $0.20$ , the strong $σ = 10$ sits high at $0.96$ . The gap between them is the whole story — Ridge trusts strong directions and distrusts weak ones. See Singular Value Decomposition.

Level 4 — Synthesis

L4.1

Derive the scalar solution from the constrained view. The parent note states that minimizing $∥ y - X w ∥^{2} + λ ∥ w ∥^{2}$ is equivalent to minimizing the fit subject to a budget $∥ w ∥^{2} \leq t$ , where $t \geq 0$ is the largest squared weight-length the model is allowed to spend (a smaller $t$ means a tighter leash). For the 1D case with fit term $s w^{2} - 2 c w + const$ (where $c = \sum x_{i} y_{i}, s = \sum x_{i}^{2}$ ) and budget $∣ w ∣ \leq t$ , show that if the OLS optimum $c / s$ violates the budget, the constrained optimum sits exactly on the boundary $w = t$ (assume $c > 0$ ), and show how a specific $λ$ produces that same $w$ .

Recall Solution

First, what is $t$ ? It is the weight budget: the constrained problem says "minimize the fit, but you may only spend up to $∥ w ∥^{2} \leq t$ of squared weight-length." In 1D, $∥ w ∥^{2} = w^{2} \leq t$ means $∣ w ∣ \leq t$ — the weight is boxed into the interval $[- t, t]$ . The unconstrained fit minimizer is $w_{OLS} = c / s$ (set $2 s w - 2 c = 0$ ). The fit term $g (w) = s w^{2} - 2 c w$ is a convex parabola with $s > 0$ , so it decreases as $w$ moves from $- \infty$ up to $w_{OLS}$ , then increases. If $w_{OLS} = c / s > t$ , then over the allowed interval $[- t, t]$ the parabola is still on its decreasing side (we never reach the vertex). The smallest fit inside the budget is therefore at the largest allowed $w$ , i.e. $w = t$ . The constraint is active (binding). How $λ$ matches $t$ . The penalized problem gives $w (λ) = c / (s + λ)$ (from L2.1). Set this equal to the boundary value $t$ and solve for the matching $λ$ : $\frac{c}{s + λ} = t \Rightarrow s + λ = \frac{c}{t} \Rightarrow λ = \frac{c}{t} - s .$ Since $c / s > t$ means $c / t > s$ , this $λ$ is positive — exactly the regime where the budget binds. So each budget $t$ corresponds to one penalty $λ = c / t - s$ , and both problems return the same shrunken weight $w = t$ . That is the concrete "two faces of one coin" correspondence: tighter budget (smaller $t$ ) ⟺ heavier penalty (larger $λ$ ).

L4.2

Ridge meets logistic. Plain Logistic Regression can drive weights to $\pm \infty$ when the classes are perfectly separable (the loss keeps improving as weights grow). Explain in one paragraph why adding an L2 penalty $λ ∥ w ∥^{2}$ fixes this, and what it does to the decision boundary's "confidence".

Recall Solution

With perfect separation, the logistic (log-loss) objective has no finite minimizer — pushing $∥ w ∥ \to \infty$ makes the sigmoid outputs saturate to exactly $0/1$ and the loss keeps dropping toward zero without ever reaching it. Adding $λ ∥ w ∥^{2}$ inserts a term that grows without bound as $∥ w ∥ \to \infty$ . Now the total objective (log-loss + penalty) does have a finite minimizer: the fit still wants to grow the weights, but the penalty fee eventually outweighs the shrinking loss gain, so growth stops at a finite $w$ . Effect on the boundary: the location of the boundary stays sensible, but its slope/confidence is tempered — because the weight magnitudes are held finite, the sigmoid no longer saturates instantly, so predicted probabilities near the boundary stay soft (not slammed to $0$ or $1$ ). Softer, calibrated probabilities generalize better on unseen data. This is the classification cousin of Ridge curing the ill-posed OLS problem: in both cases the $λ ∥ w ∥^{2}$ term supplies the missing "floor" that makes a runaway optimization settle at a finite, stable answer.

L4.3

Combine the pieces. You have $d = 500$ features but only $n = 100$ training examples. (a) Why is plain OLS impossible here? (b) Why does Ridge still return a unique answer? (c) Name one other estimator from your vault that would also give a sparse solution and say when you'd prefer it.

Recall Solution

(a) With $d > n$ , the matrix $X^{⊤} X$ (size $500 \times 500$ ) has rank at most $n = 100 < 500$ , so it is singular — not invertible — and OLS $(X^{⊤} X)^{- 1} X^{⊤} y$ does not exist (infinitely many perfect fits). (b) Ridge solves $(X^{⊤} X + λ I)^{- 1} X^{⊤} y$ . Adding $λ I$ ( $λ > 0$ ) lifts all eigenvalues above $0$ , so the matrix is strictly positive definite → invertible → unique solution, even when $d ≫ n$ . (c) Lasso (L1) would give a sparse weight vector (many exact zeros) — prefer it when you believe only a handful of the 500 features truly matter and you want automatic feature selection. If you want both grouping of correlated features (Ridge-like) and sparsity (Lasso-like), use Elastic Net.

Level 5 — Mastery

L5.1

Effective degrees of freedom. The "effective number of parameters" Ridge uses is $df (λ) = \sum_{j} \frac{σ _{j}^{2}}{σ _{j}^{2} + λ}$ (sum of the SVD shrinkage factors). For a design with singular values $σ = (3, 1)$ , compute $df (λ)$ at $λ = 0$ and $λ = 9$ , and interpret.

Recall Solution

$σ_{1}^{2} = 9, σ_{2}^{2} = 1$ .

$λ = 0$ : $df = \frac{9}{9} + \frac{1}{1} = 1 + 1 = 2$ — full model, both directions counted (this is the OLS parameter count).
$λ = 9$ : $df = \frac{9}{9 + 9} + \frac{1}{1 + 9} = \frac{9}{18} + \frac{1}{10} = 0.5 + 0.1 = 0.6$ . Interpretation: at $λ = 9$ Ridge behaves like a model with only $0.6$ effective parameters — dramatically simpler than the nominal $2$ . The strong direction contributes $0.5$ , the weak one only $0.1$ . This continuous dial from $d$ down toward $0$ is precisely how Ridge slides along the Bias-Variance Tradeoff.

L5.2

Design a cross-validation experiment. You want to pick $λ$ for a Ridge model. Lay out a correct 5-fold cross-validation procedure, and explain the one preprocessing step that must happen inside each fold, not before.

Recall Solution

Procedure.

Choose a grid, e.g. $λ \in {0.01, 0.1, 1, 10, 100}$ (log-spaced).
Split the training data into 5 folds.
For each $λ$ : for each fold $k$ , train on the other 4 folds and evaluate MSE on fold $k$ ; average the 5 MSEs → $CV (λ)$ .
Pick $λ_{best} = ar g min_{λ} CV (λ)$ — the valley of the validation curve.
Refit on all training data with $λ_{best}$ ; report on the untouched test set. The inside-the-fold step: standardization (subtracting the mean, dividing by std) must be computed on the training portion of each fold only, then applied to the held-out portion. If you standardize on the full dataset first, statistics from the validation rows leak into training and your CV estimate is optimistically biased. Never pick $λ$ by training error — it decreases monotonically toward $λ = 0$ and always lies to you.

L5.3

Full mini-pipeline by hand. Given $X = [2002], y = [42],$ compute (a) the OLS weights, (b) the Ridge weights at $λ = 4$ , and (c) the ratio $\overset{w}{^}_{j}^{ridge} / \overset{w}{^}_{j}^{OLS}$ for each component. Show that the ratio equals the SVD shrinkage factor and identify $σ_{j}$ .

Recall Solution

(a) OLS. Here $X$ is diagonal so $X^{⊤} X = [4004]$ and $X^{⊤} y = [84]$ . $\hat{w}_{OLS} = (X^{⊤} X)^{- 1} X^{⊤} y = [1/4 0 0 1/4] [84] = [21] .$ (b) Ridge, $λ = 4$ . $X^{⊤} X + λ I = [8008]$ , so $\hat{w}_{ridge} = [1/8 0 0 1/8] [84] = [1 0.5] .$ (c) Ratios. $\overset{w}{^}_{1}^{ridge} / \overset{w}{^}_{1}^{OLS} = 1/2 = 0.5$ and $\overset{w}{^}_{2}^{ridge} / \overset{w}{^}_{2}^{OLS} = 0.5/1 = 0.5$ . Both are $0.5$ . SVD check. $X = 2 I$ , so its singular values are both $σ = 2$ , giving $σ^{2} = 4$ . The shrinkage factor is $\frac{σ ^{2}}{σ ^{2} + λ} = \frac{4}{4 + 4} = 0.5$ — matching the observed ratio exactly. The SVD story ("scale each direction by $σ_{j}^{2} / (σ_{j}^{2} + λ)$ ") is not an abstraction; it is what the closed form does, component by component.

Recall One-line summaries to self-quiz

Scalar Ridge weight ::: $w = \frac{\sum x _{i} y _{i}}{\sum x _{i}^{2} + λ}$ — Ridge only inflates the denominator Why $d > n$ breaks OLS but not Ridge ::: $X^{⊤} X$ is singular; $+ λ I$ makes it invertible Ridge's choice among correlated twins ::: even split (min $\sum w_{j}^{2}$ ), e.g. $(1, 1)$ not $(50, - 48)$ SVD shrinkage factor ::: $σ_{j}^{2} / (σ_{j}^{2} + λ)$ ; weak directions shrunk most Effective degrees of freedom ::: $\sum_{j} σ_{j}^{2} / (σ_{j}^{2} + λ)$ , slides from $d$ toward $0$ Standardization in CV ::: fit scaler inside each fold, never on the full set