Intuition The core idea in one breathPlain least-squares can make weights huge to chase noise in the training data. Ridge says: "Fine, fit the data — but I will charge you a fee for every unit of weight-size you use." That fee is proportional to ∑ w j 2 \sum w_j^2 ∑ w j 2 smaller, calmer weights , which generalize better.
Definition Ridge regressionRidge regression = ordinary least squares plus a penalty on the squared L2 norm of the weights. We minimize
J ( w ) = ∑ i = 1 n ( y i − w ⊤ x i ) 2 ⏟ data fit + λ ∑ j = 1 d w j 2 ⏟ penalty J(\mathbf{w}) = \underbrace{\sum_{i=1}^{n}\big(y_i - \mathbf{w}^\top \mathbf{x}_i\big)^2}_{\text{data fit}} \;+\; \underbrace{\lambda \sum_{j=1}^{d} w_j^2}_{\text{penalty}} J ( w ) = data fit i = 1 ∑ n ( y i − w ⊤ x i ) 2 + penalty λ j = 1 ∑ d w j 2 λ ≥ 0 \lambda \ge 0 λ ≥ 0 regularization strength . The bias term w 0 w_0 w 0 not penalized.
λ = 0 \lambda = 0 λ = 0 λ → ∞ \lambda \to \infty λ → ∞ 0 0 0
Intuition Bias–variance tradeBig weights = high variance : the model reacts violently to small data changes. Shrinking weights raises bias slightly but cuts variance a lot. Since test error ≈ bias 2 + variance + noise \approx \text{bias}^2 + \text{variance} + \text{noise} ≈ bias 2 + variance + noise net win .
Intuition Why "squared" weights and not just fit?When features are correlated , the least-squares solution can put a giant positive weight on one and a giant negative weight on its twin — they cancel on training data but explode on new data. The ∑ w j 2 \sum w_j^2 ∑ w j 2 spreads weight across correlated features instead.
We work in matrix form. Let X ∈ R n × d X\in\mathbb{R}^{n\times d} X ∈ R n × d y ∈ R n \mathbf{y}\in\mathbb{R}^n y ∈ R n w ∈ R d \mathbf{w}\in\mathbb{R}^d w ∈ R d
Step 1 — Write the cost.
J ( w ) = ( y − X w ) ⊤ ( y − X w ) + λ w ⊤ w J(\mathbf{w}) = (\mathbf{y}-X\mathbf{w})^\top(\mathbf{y}-X\mathbf{w}) + \lambda\,\mathbf{w}^\top\mathbf{w} J ( w ) = ( y − X w ) ⊤ ( y − X w ) + λ w ⊤ w Why this step? ∥ y − X w ∥ 2 = ( y − X w ) ⊤ ( y − X w ) \|\mathbf{y}-X\mathbf{w}\|^2 = (\mathbf{y}-X\mathbf{w})^\top(\mathbf{y}-X\mathbf{w}) ∥ y − X w ∥ 2 = ( y − X w ) ⊤ ( y − X w ) ∑ w j 2 = w ⊤ w \sum w_j^2 = \mathbf{w}^\top\mathbf{w} ∑ w j 2 = w ⊤ w
Step 2 — Expand.
J = y ⊤ y − 2 w ⊤ X ⊤ y + w ⊤ X ⊤ X w + λ w ⊤ w J = \mathbf{y}^\top\mathbf{y} - 2\mathbf{w}^\top X^\top \mathbf{y} + \mathbf{w}^\top X^\top X \mathbf{w} + \lambda \mathbf{w}^\top\mathbf{w} J = y ⊤ y − 2 w ⊤ X ⊤ y + w ⊤ X ⊤ X w + λ w ⊤ w Why this step? Expanding lets us apply standard vector-derivative rules to each term.
Step 3 — Differentiate and set to zero.
Using ∂ ∂ w ( w ⊤ A w ) = 2 A w \frac{\partial}{\partial \mathbf{w}}(\mathbf{w}^\top A \mathbf{w}) = 2A\mathbf{w} ∂ w ∂ ( w ⊤ A w ) = 2 A w A A A ∂ ∂ w ( w ⊤ b ) = b \frac{\partial}{\partial \mathbf{w}}(\mathbf{w}^\top \mathbf{b}) = \mathbf{b} ∂ w ∂ ( w ⊤ b ) = b ∇ w J = − 2 X ⊤ y + 2 X ⊤ X w + 2 λ w = 0 \nabla_{\mathbf{w}} J = -2X^\top\mathbf{y} + 2X^\top X\mathbf{w} + 2\lambda\mathbf{w} = 0 ∇ w J = − 2 X ⊤ y + 2 X ⊤ X w + 2 λ w = 0 Why this step? At the minimum the gradient vanishes; the cost is convex (sum of convex quadratics), so this stationary point is the global minimum.
Step 4 — Solve.
( X ⊤ X + λ I ) w = X ⊤ y (X^\top X + \lambda I)\,\mathbf{w} = X^\top \mathbf{y} ( X ⊤ X + λ I ) w = X ⊤ y
+ λ I +\lambda I + λ I X ⊤ X X^\top X X ⊤ X singular (e.g. d > n d>n d > n λ I \lambda I λ I λ > 0 \lambda>0 λ > 0 X ⊤ X + λ I X^\top X + \lambda I X ⊤ X + λ I strictly positive definite → always invertible . Ridge cures the ill-posed OLS problem. This is literally where the name "ridge" comes from — you add a ridge λ \lambda λ
Let X = U Σ V ⊤ X = U\Sigma V^\top X = U Σ V ⊤ σ j \sigma_j σ j y ^ = ∑ j u j σ j 2 σ j 2 + λ u j ⊤ y . \hat{\mathbf{y}} = \sum_j \mathbf{u}_j \,\frac{\sigma_j^2}{\sigma_j^2 + \lambda}\, \mathbf{u}_j^\top \mathbf{y}. y ^ = ∑ j u j σ j 2 + λ σ j 2 u j ⊤ y .
Intuition What the factorσ j 2 σ j 2 + λ \frac{\sigma_j^2}{\sigma_j^2+\lambda} σ j 2 + λ σ j 2 large σ j \sigma_j σ j ≈ 1 \approx 1 ≈ 1 small σ j \sigma_j σ j ≈ 0 \approx 0 ≈ 0 never sets a weight exactly to 0 0 0
Worked example 1D scalar Ridge (build intuition by hand)One feature, penalty λ \lambda λ J ( w ) = ∑ i ( y i − w x i ) 2 + λ w 2 J(w)=\sum_i (y_i - w x_i)^2 + \lambda w^2 J ( w ) = ∑ i ( y i − w x i ) 2 + λ w 2 − 2 ∑ x i y i + 2 w ∑ x i 2 + 2 λ w = 0 -2\sum x_i y_i + 2w\sum x_i^2 + 2\lambda w = 0 − 2 ∑ x i y i + 2 w ∑ x i 2 + 2 λ w = 0 w = ∑ x i y i ∑ x i 2 + λ . w = \frac{\sum x_i y_i}{\sum x_i^2 + \lambda}. w = ∑ x i 2 + λ ∑ x i y i . Why this step? The OLS answer is ∑ x i y i ∑ x i 2 \frac{\sum x_i y_i}{\sum x_i^2} ∑ x i 2 ∑ x i y i inflates the denominator by λ \lambda λ ∣ w ∣ |w| ∣ w ∣ λ → ∞ \lambda\to\infty λ → ∞ w → 0 w\to 0 w → 0 ∑ x i y i = 10 \sum x_iy_i=10 ∑ x i y i = 10 ∑ x i 2 = 5 \sum x_i^2=5 ∑ x i 2 = 5 w = 2 w=2 w = 2 λ = 5 \lambda=5 λ = 5 w = 10 / 10 = 1 w=10/10=1 w = 10/10 = 1
Worked example Reading a validation curveYou sweep λ ∈ { 0 , 0.1 , 1 , 10 , 100 } \lambda \in \{0, 0.1, 1, 10, 100\} λ ∈ { 0 , 0.1 , 1 , 10 , 100 } λ = 0 \lambda=0 λ = 0 λ = 1 \lambda=1 λ = 1 λ = 100 \lambda=100 λ = 100 Pick λ \lambda λ via cross-validation.
Why this step? λ \lambda λ λ \lambda λ
Common mistake "Ridge produces sparse models / feature selection."Why it feels right: regularization "removes" complexity, so it seems it should zero-out useless features.
Fix: L2 shrinks weights smoothly toward 0 0 0 exactly to 0 0 0 L1 (Lasso) whose diamond-shaped constraint has corners on the axes. Ridge's circular constraint has no corners.
Common mistake Forgetting to standardize features first.Why it feels right: "the math works on raw numbers."
Fix: λ ∑ w j 2 \lambda\sum w_j^2 λ ∑ w j 2 1000 × 1000\times 1000 × standardize (zero mean, unit variance) so the penalty is fair across features.
Common mistake Penalizing the bias/intercept.Why it feels right: "w 0 w_0 w 0 Fix: The intercept just sets the output baseline; shrinking it biases predictions toward 0 0 0 w 0 w_0 w 0 y \mathbf{y} y
λ \lambda λ Why it feels right: we minimize a cost, so minimize it fully.
Fix: Training error is monotonic in λ \lambda λ λ = 0 \lambda=0 λ = 0 cross-validation on held-out data.
Intuition Two faces of the same coinMinimizing ∥ y − X w ∥ 2 + λ ∥ w ∥ 2 \|\mathbf{y}-X\mathbf{w}\|^2 + \lambda\|\mathbf{w}\|^2 ∥ y − X w ∥ 2 + λ ∥ w ∥ 2 min ∥ y − X w ∥ 2 subject to ∥ w ∥ 2 ≤ t . \min \|\mathbf{y}-X\mathbf{w}\|^2 \quad \text{subject to} \quad \|\mathbf{w}\|^2 \le t. min ∥ y − X w ∥ 2 subject to ∥ w ∥ 2 ≤ t . λ ⇔ \lambda \Leftrightarrow λ ⇔ t t t ball of radius t \sqrt{t} t
Ridge cost function ∑ i ( y i − w ⊤ x i ) 2 + λ ∑ j w j 2 \sum_i (y_i-\mathbf{w}^\top\mathbf{x}_i)^2 + \lambda\sum_j w_j^2 ∑ i ( y i − w ⊤ x i ) 2 + λ ∑ j w j 2 Ridge closed-form solution w ^ = ( X ⊤ X + λ I ) − 1 X ⊤ y \hat{\mathbf{w}}=(X^\top X+\lambda I)^{-1}X^\top\mathbf{y} w ^ = ( X ⊤ X + λ I ) − 1 X ⊤ y Why add λ I \lambda I λ I Makes
X ⊤ X + λ I X^\top X+\lambda I X ⊤ X + λ I invertible/positive-definite even when
X ⊤ X X^\top X X ⊤ X is singular
What happens as λ → ∞ \lambda\to\infty λ → ∞ All weights shrink toward
0 0 0 (max bias, min variance, underfit)
What happens at λ = 0 \lambda=0 λ = 0 Recovers ordinary least squares
Does Ridge give sparse weights? No — it shrinks smoothly but rarely zeros weights; that's Lasso (L1)
How does Ridge treat correlated features? Spreads weight evenly among them (smallest-norm split) instead of huge cancelling values
Bias–variance effect of increasing λ \lambda λ Increases bias, decreases variance
How should λ \lambda λ Cross-validation on held-out data, not training error
Preprocessing needed before Ridge Standardize features so the penalty is scale-fair
Should the intercept be penalized? No — center the data / exclude
w 0 w_0 w 0 from the penalty
SVD shrinkage factor per direction σ j 2 / ( σ j 2 + λ ) \sigma_j^2/(\sigma_j^2+\lambda) σ j 2 / ( σ j 2 + λ ) — keeps strong directions, shrinks weak ones
1D Ridge weight formula w = ∑ x i y i ∑ x i 2 + λ w=\frac{\sum x_iy_i}{\sum x_i^2+\lambda} w = ∑ x i 2 + λ ∑ x i y i Constraint-form equivalent of Ridge Minimize
∥ y − X w ∥ 2 \|\mathbf{y}-X\mathbf{w}\|^2 ∥ y − X w ∥ 2 s.t.
∥ w ∥ 2 ≤ t \|\mathbf{w}\|^2\le t ∥ w ∥ 2 ≤ t
Recall Feynman: explain to a 12-year-oldImagine building a paper tower to reach a mark. You could stack a crazy tall wobbly tower that just barely touches the mark — but it falls over if the wind (new data) blows. Ridge is a rule: "Every extra block you use costs money." So you build the shortest, sturdiest tower that still reaches close to the mark. It won't be perfect, but it won't topple when things change. The "cost per block" is λ \lambda λ
"L2 = squares, spreads, shrinks (but never kills)." And the fix "+ λ I +\lambda I + λ I ridge along the diagonal so the matrix can always be inverted. L2 → penalty is squared (power 2 ); L1 → Lasso, cuts to zero.
charges fee for weight size
Penalty lambda sum w squared
Lambda regularization strength
Closed form w = X^T X + lambda I inverse X^T y
Intuition Hinglish mein samjhoDekho, plain least-squares regression ka ek problem hai: agar data me thoda noise ho ya features aapas me correlated ho, to model bahut bade-bade weights bana leta hai training data ko exactly fit karne ke chakkar me. Ye weights new data pe faad khaate hain — yani overfitting. Ridge (L2) regularization is problem ka ilaaj hai: hum cost function me ek extra "fee" jodte hain, λ ∑ w j 2 \lambda \sum w_j^2 λ ∑ w j 2
Formula seedha derive hota hai: cost ko w \mathbf{w} w w ^ = ( X ⊤ X + λ I ) − 1 X ⊤ y \hat{\mathbf{w}} = (X^\top X + \lambda I)^{-1} X^\top \mathbf{y} w ^ = ( X ⊤ X + λ I ) − 1 X ⊤ y λ I \lambda I λ I X ⊤ X X^\top X X ⊤ X + λ I +\lambda I + λ I
Do cheezein yaad rakhna. Pehli — Ridge weights ko shrink karta hai, exactly zero nahi karta; sparsity chahiye to Lasso (L1) use karo. Dusri — λ \lambda λ cross-validation karo, training error se nahi, kyunki training error hamesha λ = 0 \lambda=0 λ = 0 standardize karo (zero mean, unit variance), warna meter vs millimeter wale features ko galat penalty milegi.
Simple analogy: paper ka tower banana hai jo ek mark tak pahunche. Bina rule ke tum ek lambi wobbly tower bana loge jo hawa aane pe gir jayegi. Ridge kehta hai "har block ka paisa lagega" — to tum sabse chhoti, mazboot tower banaoge jo hawa (new data) me bhi khadi rahe. λ \lambda λ
