Foundations — Regularization — L1 (lasso), L2 (ridge), dropout

5.6.3 · D1 · Coding › Machine Learning (Aerospace Applications) › Regularization — L1 (lasso), L2 (ridge), dropout

Parent note ko — jo L1, L2, aur dropout par hai — padhne se pehle, tumhe usmein use hone wale har symbol ko bina rukke padhna aana chahiye. Yeh page har ek cheez ko bilkul zero se build karta hai, us order mein jisme woh ek doosre par depend karte hain.

1. A weight $w$ — the knob

Ek mixing desk ki picture socho. Har slider ek $w_i$ hai. Slider $i$ ko upar karna matlab hai "input number $i$ par zyada dhyan do." Saare sliders ke collection ko bold $\mathbf{w}$ likhte hain (yeh ek vector hai — numbers ki ek ordered list).

Parent ke model mein jo letter $b$ aata hai ($C_D=w_1\alpha +\cdots +b$) woh bias hai — ek extra slider jo kisi bhi input se independent, har cheez ko ek constant amount se upar ya neeche shift karta hai. Ise usually penalize nahi kiya jaata.

2. Prediction $\hat{y}$ vs. truth $y$ — the hat means "guess"

Number line par do dots ki picture socho: true value $y$ par ek kaala dot, aur guess $\hat{y}$ par ek hollow dot. Unke beech ka gap is ek example par model ki galti hai.

3. Summation $\sum$ — "add up a list"

Ek shopping receipt ki picture socho: har line ek $a_n$ hai, aur $\sum$ neeche ka total hai.

Topic ko $\sum$ isliye chahiye kyunki ek penalty ko saare weights ko ek single number mein combine karna hota hai taaki score mein add kiya ja sake — tum sliders ke poore desk ko unhe sum kiye bina penalize nahi kar sakte.

4. The loss $\mathcal{L}$ — the badness score

Ek valley ki picture socho. Horizontal position tumhari weights ki setting hai; zameen ki height loss hai. Training ek ball ko lowest point ki taraf neeche roll karna hai.

Parent note loss ko do named pieces mein split karta hai:

Regularization ka poora point yeh hai: sirf ${\mathcal{L}}_{\text{data}}$ minimize mat karo; ${\mathcal{L}}_{\text{total}}$ minimize karo, jo knob size ki bhi parwah karta hai.

Sum ke andar chhota $\ell (y_n,{\hat{y}}_n)$ per-example loss hai — sirf ek flight $n$ par badness. Un sab ko sum karke $N$ (examples ki sankhya) se divide karne par average milta hai, jo ${\mathcal{L}}_{\text{data}}$ hai.

5. Magnitude bars: $|w|$ and $\Vert w\Vert$ — "how big, ignore direction"

Map par point $w=(w_1,w_2)$ ki picture socho. $\Vert w{\Vert }_2$ origin tak crow-flies distance hai. $\Vert w{\Vert }_1$ taxi-cab distance hai (tumhe grid streets par travel karna hoga).

6. The slope $\frac{\partial \mathcal{L}}{\partial w_i}$ — which way is downhill

Hume ek derivative chahiye (aur loss ko sirf evaluate karna nahi) kyunki neeche roll karne ke liye tumhe apne current spot par down ka direction pata hona chahiye — woh direction exactly woh slope hai.

Curly $\partial$ (straight $d$ ki jagah) ek flag hai: "kai saare knobs hain; main sirf isko change kar raha hoon aur baaki ko still rakh raha hoon."

Saare weights ke liye saath stacked partials gradient banate hain, likha jaata hai $\nabla \mathcal{L}$ (ulta triangle "nabla"). Yeh sirf saare slopes ki poori list hai — complete "downhill arrow."

7. The gradient-descent step: $\eta$, $\lambda$, aur arrow $\leftarrow$

$w^{(t)}$ mein superscript ka matlab sirf "step number $t$ par weight" hai — $t=0$ start hai, $t=1$ ek update ke baad, aur aage bhi. Yeh ek time stamp hai, power nahi.

8. Probability $p$ aur expectation $\mathbb{E}$ — dropout ki language

Parent mein "$m_i\sim \text{Bernoulli}(1-p)$" ka matlab hai: $m_i$ ek random on/off switch hai jo probability $1-p$ ke saath $1$ hai aur otherwise $0$ hai. $h_i$ ko $m_i$ se multiply karna us neuron ko keep ya kill karta hai.

Prerequisite map

Yeh seedha parent topic mein jaata hai. Related roads: Gradient Descent Variants update rule ko refine karta hai, Feature Engineering aur Overfitting Detection motivate karte hain kyun hum penalize karte hain, Cross-Validation $\lambda$ choose karta hai, aur Ensemble Methods plus Bayesian Inference dropout ke kaam karne ki deeper wajahein dete hain.

Equipment checklist