Foundations — L2 (Ridge) regularization

This page assumes nothing. Before we can even read the Ridge cost function, we must earn every letter in it: what a weight is, what a vector is, what $\sum$ means, what a transpose does, what a matrix inverse is. Each symbol gets a plain-words meaning, a picture, and a reason the topic needs it. Read top to bottom — every idea rests on the one before.

1. A "feature" and a "weight" and the answer — the atoms

So there are two different $y$'s to keep straight: $y$ is the truth (given in the data), $\hat{y}$ is our guess (computed from weights). The whole game is making $\hat{y}$ close to $y$.

Picture a straight line through the origin. The weight $w$ is its slope — how steeply the prediction climbs as the feature grows.

Why the topic needs this: Ridge is entirely a story about keeping $w$ small while $\hat{y}$ stays close to $y$. If you don't feel $w$ as a slope-dial that can grow dangerously large, the whole "charge a fee for big weights" idea has nothing to grab onto. Look at the red steep line in the figure — that is what "huge weight chasing noise" looks like.

2. The intercept $w_0$ — the line that need not pass through the origin

The line in Section 1 was forced through the origin ($\hat{y}=wx$, so $x=0\Rightarrow \hat{y}=0$). Real data rarely obeys that: even a house of "zero extra size" has some baseline price. We fix this with one more number.

Why the topic needs this: The parent note says "the bias term $w_0$ is usually not penalized." You cannot understand that sentence without knowing what $w_0$ is and why it is special. From here on, assume the data has been centered, so we focus purely on the slope-weights $\mathbf{w}=(w_1,\dots ,w_d)$.

3. Many features at once — the vector $\mathbf{x}$ and $\mathbf{w}$

Real problems have many features per example (size and age and location...). We stack the features of one example into a list of numbers called a vector, written in bold: $\mathbf{x}=(x_1,x_2,\dots ,x_d)$. Each $x_j$ is feature number $j$, and $d$ is how many features there are.

The weights get their own vector $\mathbf{w}=(w_1,\dots ,w_d)$ — one dial per feature.

Picture $\mathbf{w}$ as an arrow in space: in 2 features it's an arrow on a flat page, its two coordinates being $w_1$ (rightward) and $w_2$ (upward). The length of this arrow will turn out to be exactly what Ridge charges you for.

Why the topic needs this: Ridge's penalty is "keep the weight arrow short." You cannot picture "short arrow" without first picturing the arrow.

4. Summation $\sum$ — "add up over all slots"

The parent note is full of $\sum$. It is just shorthand for repeated addition.

Two uses appear constantly:

• Over examples $i=1\dots n$ (where $n$ = number of training rows). Here $y_i$ = the true answer of the $i$-th example, and ${\hat{y}}_i$ = our prediction for it.
• Over features $j=1\dots d$.

Why the topic needs this: The data-fit term sums an error over all examples (${\sum }_i$); the penalty sums squared weights over all features (${\sum }_j$). Mixing them up breaks everything.

5. The squared error and the L2 norm — "size measured with squares"

Error of one prediction: using the symbols from Section 1, the error on example $i$ is $y_i-{\hat{y}}_i$ — the gap between the true answer $y_i$ and our guess ${\hat{y}}_i$. We square it, $(y_i-{\hat{y}}_i{)}^2$, for two reasons: squaring makes it positive (a miss of $-3$ is as bad as $+3$), and it punishes big misses much harder than small ones.

Now the star of the show:

Picture the right triangle whose legs are $w_1$ and $w_2$: the hypotenuse is $\Vert \mathbf{w}\Vert$, the arrow length. Squaring it is measuring the squared radius of a circle the arrow lives on.

Why the topic needs this: The name "L2" in "L2 regularization" is this norm. The penalty $\lambda \sum w_j^2$ is $\lambda \Vert \mathbf{w}{\Vert }^2$ — literally "$\lambda$ times the squared arrow length." Look at the figure: Ridge charges you the squared radius, so it hates long arrows enormously.

6. The dot product ${\mathbf{w}}^{\top }\mathbf{x}$ — the full prediction

With many features (and data centered so $w_0=0$), the prediction is feature-by-feature multiply, then add: $\hat{y}=w_1{x}_1+w_2{x}_2+\cdots +w_d{x}_d={\sum }_{j=1}^d{w}_j{x}_j.$

This pattern is so common it has a name: the dot product, written ${\mathbf{w}}^{\top }\mathbf{x}$.

Why the topic needs this: Every appearance of ${\mathbf{w}}^{\top }{\mathbf{x}}_i$ in the parent is "the model's prediction ${\hat{y}}_i$ for example $i$." It's just ${\sum }_j{w}_j{x}_{ij}$ dressed in compact clothing.

7. The set $\mathbb{R}$ and stacking data into a matrix $X$

Before we build the data table, one tiny piece of notation.

We have $n$ examples, each a feature-vector. Stack them as rows of a table — that table is the design matrix $X\in {\mathbb{R}}^{n\times d}$, with $n$ rows and $d$ columns.

Picture $X$ as a spreadsheet, $\mathbf{y}$ as the answer column beside it, and $X\mathbf{w}$ as the column of guesses you'd write next to them.

Why the topic needs this: The closed-form solution $\hat{\mathbf{w}}=(X^{\top }X+\lambda I{)}^{-1}{X}^{\top }\mathbf{y}$ is written entirely in these matrix pieces. $\mathbf{y}-X\mathbf{w}$ is the column of all errors; $\Vert \mathbf{y}-X\mathbf{w}{\Vert }^2$ is the total squared error ${\sum }_i(y_i-{\mathbf{w}}^{\top }{\mathbf{x}}_i{)}^2$, using the row ${\mathbf{x}}_i$ just defined.

8. The identity matrix $I$ and matrix inverse — undoing multiplication

To solve for $\mathbf{w}$ we must "divide" by a matrix. Matrices don't divide; they invert.

Why the topic needs this: The whole closed form hinges on $(X^{\top }X+\lambda I{)}^{-1}$ existing. This is the single most important thing $\lambda$ buys you beyond shrinkage.

9. The lever $\lambda$ — the fee rate

Picture a slider: push it right and the weight-arrow is forced shorter and shorter.

Why the topic needs this: $\lambda$ is the one dial the practitioner sets. Everything Ridge does — cure singularity, trade bias for variance, shrink noisy directions — is controlled by it. Look at how the arrow collapses toward the origin as $\lambda$ grows.

10. The cost $J$, its gradient, and convexity — why "derivative $=0$" works

Everything so far combines into a single number we want to make small: the cost (or loss) $J(\mathbf{w})$.

To find the smallest-$J$ weight, we use two more ideas.

Why the topic needs this: The Ridge cost $J$ is a sum of squares plus $\lambda \Vert \mathbf{w}{\Vert }^2$ — a perfect bowl. That is why solving ${\nabla }_{\mathbf{w}}J=0$ gives the answer directly, and why a unique closed form even exists.

Prerequisite map

Read the map as: the atoms (feature, weight, answer) split into the intercept story on one side and the weight-arrow story on the other; the arrow gives us length (the penalty) and dot product (the prediction); those build the data matrix, the cost, and finally the closed form. From here, continue to the parent L2 (Ridge) regularization, and lean on Ordinary Least Squares (the $\lambda =0$ case), Feature Scaling / Standardization (so the penalty is fair), Singular Value Decomposition (the deep shrinkage view), Bias-Variance Tradeoff and Cross-Validation (choosing $\lambda$), while L1 (Lasso) regularization and Elastic Net contrast the penalty shape, and Logistic Regression reuses the same weight-shrinking trick.

Equipment checklist

Test yourself — reveal only after you've answered aloud.