L2 (Ridge) regularization — Hinglish notes

2.2.14 · AI-ML › Linear & Logistic Regression

YEH HAI KYA?

$λ = 0$ → ordinary least squares (koi shrinkage nahi).
$λ \to \infty$ → saare weights $0$ ki taraf force ho jaate hain (underfit).

YEH HELP KYUN KARTA HAI? (Reason ko steel-man karna)

KAISE: closed form scratch se derive karo

Hum matrix form mein kaam karte hain. Maano $X \in R^{n \times d}$ (rows = examples), $y \in R^{n}$ , weights $w \in R^{d}$ .

Step 1 — Cost likhna. $J (w) = (y - X w)^{⊤} (y - X w) + λ w^{⊤} w$ Yeh step kyun? $∥ y - X w ∥^{2} = (y - X w)^{⊤} (y - X w)$ , aur $\sum w_{j}^{2} = w^{⊤} w$ . Ab sab kuch differentiable hai.

Step 2 — Expand karo. $J = y^{⊤} y - 2 w^{⊤} X^{⊤} y + w^{⊤} X^{⊤} X w + λ w^{⊤} w$ Yeh step kyun? Expand karne se hum har term par standard vector-derivative rules apply kar sakte hain.

Step 3 — Differentiate karo aur zero set karo. $\frac{\partial}{\partial w} (w^{⊤} A w) = 2 A w$ (symmetric $A$ ke liye) aur $\frac{\partial}{\partial w} (w^{⊤} b) = b$ use karte hue: $\nabla_{w} J = - 2 X^{⊤} y + 2 X^{⊤} X w + 2 λ w = 0$ Yeh step kyun? Minimum par gradient zero hota hai; cost convex hai (convex quadratics ka sum), isliye yeh stationary point global minimum hai.

Step 4 — Solve karo. $(X^{⊤} X + λ I) w = X^{⊤} y$

SVD ke through Shrinkage (deep view)

Maano $X = U Σ V^{⊤}$ (SVD), singular values $σ_{j}$ ke saath. Tab dikhaaya ja sakta hai ki prediction hai $\hat{y} = \sum_{j} u_{j} \frac{σ _{j}^{2}}{σ _{j}^{2} + λ} u_{j}^{⊤} y .$

Worked examples

Common mistakes

Constrained-optimization view

Flashcards

Ridge cost function

\sum_{i} (y_{i} - w^{⊤} x_{i})^{2} + λ \sum_{j} w_{j}^{2}

Ridge closed-form solution

\hat{w} = (X^{⊤} X + λ I)^{- 1} X^{⊤} y

λ I

kyun add karte hain?

X^{⊤} X + λ I

ko invertible/positive-definite banata hai jab bhi

X^{⊤} X

singular ho

λ \to \infty

par kya hota hai?

Saare weights

0

ki taraf shrink ho jaate hain (max bias, min variance, underfit)

λ = 0

par kya hota hai?

Ordinary least squares wapas milta hai

Kya Ridge sparse weights deta hai?

Nahi — yeh smoothly shrink karta hai lekin rarely weights zero karta hai; yeh Lasso (L1) ka kaam hai

Ridge correlated features ke saath kaise treat karta hai?

Unke beech weight evenly spread karta hai (smallest-norm split) huge cancelling values ki jagah

λ

badhane ka bias–variance effect

Bias badhata hai, variance kam karta hai

λ

kaise choose karein?

Held-out data par cross-validation se, training error se nahi

Ridge se pehle zaroori preprocessing

Features standardize karo taaki penalty scale-fair ho

Kya intercept ko penalize karna chahiye?

Nahi — data center karo /

w_{0}

ko penalty se bahar rakho

SVD shrinkage factor per direction

σ_{j}^{2} / (σ_{j}^{2} + λ)

— strong directions rakhta hai, weak ones shrink karta hai

1D Ridge weight formula

w = \frac{\sum x _{i} y _{i}}{\sum x _{i}^{2} + λ}

Ridge ka constraint-form equivalent

Minimize karo

∥ y - X w ∥^{2}

s.t.

∥ w ∥^{2} \leq t

Recall Feynman: 12 saal ke bacche ko explain karo

Socho tum ek mark tak pahunchne ke liye paper tower bana rahe ho. Tum ek bahut tall aur wobbly tower bana sakte ho jo barely mark tak pahunchti hai — lekin woh girr jaayegi agar hawa (new data) chale. Ridge ek rule hai: "Har extra block use karne ka paisa lagega." Toh tum sabse chhoti, mazboot tower banate ho jo phir bhi mark ke kaafi paas pahunche. Woh perfect nahi hogi, lekin cheezon ke badalne par giregi nahi. "Cost per block" hai $λ$ : kuch bhi charge mat karo aur tum ek monster banaate ho; bahut zyada charge karo aur tum almost kuch nahi banaate.

Connections

Ordinary Least Squares — $λ = 0$ ke saath Ridge.
L1 (Lasso) regularization — sibling penalty, sparsity deta hai.
Elastic Net — L1 + L2 combine karta hai.
Bias-Variance Tradeoff — reason kyon Ridge help karta hai.
Singular Value Decomposition — directional shrinkage reveal karta hai.
Cross-Validation — $λ$ kaise choose karein.
Feature Scaling / Standardization — zaroori preprocessing.
Logistic Regression — L2 penalty wahan bhi apply hoti hai (weight decay).