Regularization — L1 (lasso), L2 (ridge), dropout: Question bank

5.6.3 · D5 · Coding › Machine Learning (Aerospace Applications) › Regularization — L1 (lasso), L2 (ridge), dropout

Shuru karne se pehle, ek shared vocabulary reminder taaki har reveal clearly samajh aaye:

Definition Is page par use kiye gaye symbols

$w_{i}$ — ek single weight (ek number jo model tune karta hai). "Small weights" = simpler, smoother model.
$λ$ (lambda) — regularization strength ka knob. Bada $λ$ = complexity par zyada harsh penalty.
$η$ (eta) — learning rate, yani woh step size jo gradient descent leta hai.
$p$ — dropout probability, woh chance ki ek neuron ek training pass ke dauran switch off ho jaaye.
$h_{i}$ — hidden neuron $i$ ki clean activation (output number) — woh value jo woh emit karta agar koi dropout na hota.
$\tilde{h}_{i}$ (h with a tilde) — usi activation ka dropout-corrupted version: ya toh zero ho jaata hai (agar neuron drop hua) ya reh jaata hai (possibly rescaled).
"Sparse" = bahut saare weights exactly zero hain. "Shrunk" = weights chhote hain par nonzero hain.

Definition Do dropout conventions (inhe define karo pehle ki hum tumhe trap karein)

Neurons ko drop karne se forward flow hone wala average signal ghatta hai, isliye kahin na kahin compensate karna padta hai. Do conventions yeh alag-alag jagah karte hain:

Classic dropout: training ke dauran sirf neurons ko zero karo (koi rescale nahi). Test time par har activation ko $(1 - p)$ se multiply karo taaki shrunken training average match ho. Correction test time par hoti hai.
Inverted dropout (PyTorch/TensorFlow mein modern default): training ke dauran har kept activation ko $(1 - p)$ se divide karo, yani $\tilde{h}_{i} = h_{i} / (1 - p)$ . Isse expected activation clean $h_{i}$ ke barabar ho jaati hai, isliye test time par koi scaling bilkul nahi chahiye. Correction training time par hoti hai. Neeche jo bhi reveal "inverted dropout" kehta hai woh is doosri convention ko refer karta hai.

L1-versus-L2 geometry neeche poocha jaata hai, isliye ek baar picture se ground karo:

Figure — Regularization — L1 (lasso), L2 (ridge), dropout

Blue diamond L1 region $∣ w_{1} ∣ + ∣ w_{2} ∣ \leq C$ hai; yellow circle L2 region $w_{1}^{2} + w_{2}^{2} \leq C$ hai. Red ellipses loss contours hain jo unconstrained best fit ki taraf shrink ho rahi hain. Notice karo ki diamond pehle apne sharp corner pe $w_{1}$ -axis par touch hoti hai, jahan $w_{2} = 0$ exactly hai — yeh ek sparse solution hai. Smooth circle apni side par touch hoti hai, aisi jagah jahan dono weights nonzero hain. Is image ko in-built rakhna "Why questions" section ke liye.

True or false — justify karo

L2 regularization bahut saare weights ko exactly zero kar deta hai, ek sparse model deta hai.

False — L2 ki shrinkage proportional hoti hai (

w_{i} \leftarrow (1 - η λ) w_{i}

), isliye weights zero ke paas aate hain par essentially kabhi wahan land nahi karte. L1 exact zeros produce karta hai.

λ

ko bahut bada set karna generalization hamesha improve karta hai.

False — jab

λ \to \infty

toh har weight zero ki taraf crush ho jaata hai aur model underfit karta hai, real signal ko ignore karta hai. Ek sweet spot hota hai jo Cross-Validation se milta hai, koi "zyada better hai" wala rule nahi hai.

Dropout test time par bilkul training time jaisa apply hota hai.

False — test time par har neuron ko keep kiya jaata hai aur (inverted dropout ke saath) koi scaling nahi hoti; final prediction ke liye tum network ki full capacity chahte ho, koi crippled sub-network nahi.

L1 aur L2 penalties dono weights ko chhota rakhte hain, isliye yeh interchangeable hain.

False — yeh alag-alag shrink karte hain: L2 square ko penalize karta hai (huge weights sabse zyada punish hote hain, tiny ones ko barely touch kiya jaata hai), jabki L1 absolute value ko penalize karta hai (constant push jo chhote weights ko zero kar deta hai). L1 features select karta hai; L2 sirf smooth karta hai.

Regularization training accuracy reduce karta hai aur yeh ek bug hai jise avoid karna chahiye.

False — training accuracy mein thodi giravat intended trade hai: tum noise ko fit karna sacrifice karte ho taaki model naye flights par generalize kare. Lower train accuracy ke saath higher test accuracy ek success hai.

L2 penalty

\frac{λ}{2} \sum w_{i}^{2}

mein

\frac{1}{2}

model ke behaviour ko change karta hai.

False — yeh ek cosmetic convenience hai jo

w_{i}^{2}

differentiate karne se aane wale factor of 2 ko cancel kar deta hai. Tum ise drop karke sirf

λ

rescale kar sakte ho; achievable models ki family identical hai.

Inverted dropout ke saath, training ke dauran ek neuron ki expected activation uski clean activation ke barabar hoti hai.

True — jab

\tilde{h}_{i} = h_{i} / (1 - p)

kept hone par (probability

1 - p

) aur

0

dropped hone par (probability

p

), average hai

E [\tilde{h}_{i}] = (1 - p) \cdot \frac{h _{i}}{1 - p} + p \cdot 0 = h_{i}

, yahi reason hai ki test time par koi rescaling nahi chahiye.

L2 regularization aur "weight decay" ek hi cheez hai.

True (plain SGD ke liye) — L2 update ko rearrange karne par milta hai

w_{i} \leftarrow (1 - η λ) w_{i} - η \partial L_{data}

, aur woh

(1 - η λ)

factor literally data gradient apply hone se pehle har step mein weight ko decay karta hai.

Dropout ko input layer par bhi usi

p = 0.5

se use kar sakte hain jo hidden layers use karti hain.

False in practice — raw inputs ka aadha drop karna bahut zyada information destroy kar deta hai; input dropout, agar use bhi ho, toh bahut chhota

p

use karta hai (jaise 0.1–0.2).

p = 0.5

rule of thumb hidden/FC layers ke liye hai.

Error dhundho

"Regularize karne ke liye, maine loss mein

λ \sum_{i} w_{i}^{2}

add kiya aur bias term

b

mein bhi."

Error bias ko regularize karna hai. Bias sirf output level shift karta hai; ise penalize karna model ki predictions center karne ki ability se ladhta hai aur koi overfitting benefit nahi deta. Weights ko penalize karo,

b

ko free chhodo.

"L1 update:

w_{i} \leftarrow (1 - η λ) w_{i} - η \partial L_{data}

Yeh L2 update hai. L1 ek constant

η λ \cdot sign (w_{i})

subtract karta hai, proportional factor nahi:

w_{i} \leftarrow w_{i} - η (\partial L_{data} + λ sign (w_{i}))

∣ w_{i} ∣

har jagah differentiable hai, isliye L1 gradients hamesha well-defined hote hain."

Galat —

∣ w_{i} ∣

mein

w_{i} = 0

par ek kink hai jahan derivative undefined hai. Hum ise ek subgradient se patch karte hain (

[- 1, 1]

mein koi bhi value), aur wahan

0

choose karna exactly wahi hai jo weights ko zero par baithne deta hai.

"Maine test time par uncertainty estimates pane ke liye dropout use kiya, isliye activations ko

(1 - p)

ki jagah

\frac{1}{1 - p}

se scale kiya."

Conventions mix ho gayi. Agar training ke dauran already

\frac{1}{1 - p}

se scale kar liya (inverted dropout), toh test time par koi scaling ki zaroorat nahi hai. Isse dobara apply karna double-count karta hai aur activations inflate kar deta hai.

"Kyunki dropout randomly neurons zero karta hai, yeh network ko random guesser bana deta hai aur training hurt karta hai."

Randomness hi point hai, koi flaw nahi — har pass ek alag sub-network train karta hai, aur ~

2^{n}

sub-networks ka ensemble ek robust model mein average ho jaata hai (dekho Ensemble Methods). Loss noisier hoti hai par generalization improve hoti hai.

"Mera Ridge model abhi bhi overfit kar raha hai, isliye maine sign badal kar

- \frac{λ}{2} \sum w_{i}^{2}

kar diya taaki zyada penalize ho."

Negative penalty large weights ko reward karti hai — regularization ka bilkul ulta. Loss ko arbitrarily low drive kiya ja sakta hai weights blow up karke, model barbaad ho jaata hai.

Why questions

L1 diamond constraint sparse solutions kyun produce karta hai jabki L2 circle nahi karta?

Upar ki figure dekho: diamond

∣ w_{1} ∣ + ∣ w_{2} ∣ \leq C

ke sharp corners axes par hain, jahan ek coordinate zero hai. Loss contours (ellipses) aksar pehle inhi corners ko touch karti hain. L2 circle smooth hai aur aise koi corners nahi hain, isliye yeh apni side par touch hoti hai jahan dono weights nonzero rehte hain.

L2 update ki exact rearrangement

(1 - η λ)

factor kyun nikaalti hai?

w_{i} \leftarrow w_{i} - η (\partial L_{data} + λ w_{i})

se shuru karo.

η

distribute karo:

w_{i} \leftarrow w_{i} - η \partial L_{data} - η λ w_{i}

. Ab do

w_{i}

terms collect karo —

w_{i} - η λ w_{i} = (1 - η λ) w_{i}

— jo

w_{i} \leftarrow (1 - η λ) w_{i} - η \partial L_{data}

deta hai.

(1 - η λ)

purely

w_{i}

ko un do terms se factor out karne se aata hai, aur yeh dikhata hai ki data gradient kaam karne se pehle weight decay hoti hai.

Jab smooth, well-conditioned weights chahiye (feature selection nahi), toh L2 penalty prefer kyun ki jaati hai?

Squared term har jagah differentiable hai aur weights ko proportionally shrink karta hai, correlated features mein influence spread karta hai instead of kuch ko arbitrarily zero karne ke. Yeh stable, smooth solutions deta hai — useful jab sab features thoda signal carry karte hain.

Dropout ek ensemble jaisa kyun behave karta hai jabki tum sirf ek network train karte ho?

Har mini-batch ek alag random sub-network sample karta hai (neurons ka alag subset active). Training ke dauran yeh weights share karte hain par bahut saari architectures explore karte hain, isliye final network exponentially many thin networks ke average jaisa behave karta hai — Ensemble Methods effect bina bahut saare models train kiye.

L1 ko automatic feature selection kyun mana jaata hai par L2 ko nahi?

L1 uninformative weights ko exactly zero push karta hai, toh woh features model se bilkul bahar ho jaate hain — yeh ek selection decision hai. L2 har feature ko small nonzero weight ke saath rakhta hai, isliye kuch bhi truly remove nahi hota. Yeh L1 ko Feature Engineering aur interpretability se jodhta hai.

L1 ki constant-subtraction behaviour chhote weights ko vanish kyun karti hai par bade weights ko mostly intact chhod deti hai?

Har step ek fixed

η λ

remove karta hai (soft thresholding). Uss step se chhota weight zero ke paas push ho jaata hai aur zero par clamp ho jaata hai; ek bada weight utna hi fixed amount lose karta hai, jo uski size ka negligible fraction hai, isliye woh survive karta hai.

λ

(aur

p

) ko validation set se kyun tune karte hain, training loss se kyun nahi?

Training loss hamesha less regularization prefer karta hai (woh noise fit kar sakta hai). Sirf held-out data via Cross-Validation woh value reveal karta hai jo best generalize karti hai; training error par pick karna

λ \to 0

drive kar dega aur overfitting wapas laayega.

Edge cases

L2 update kya karta hai jab

λ = 0

ho?

Yeh plain gradient descent

w_{i} \leftarrow w_{i} - η \partial L_{data}

mein collapse ho jaata hai — koi decay nahi, koi penalty nahi, overfitting ka full risk.

λ = 0

matlab "regularization off."

Agar

η λ = 1

exactly ho (aur data gradient zero ho) toh har weight ka kya hoga?

Decay factor

(1 - η λ) = 0

ho jaata hai, isliye

w_{i}

ek hi step mein zero ho jaata hai. Yeh ek degenerate, too-aggressive setting hai; normally

η λ ≪ 1

rakhte hain.

w_{i} = 0

par L1 subgradient kya hai, aur allowed range

[- 1, + 1]

mein se specifically

0

kyun pick karte hain?

Classical derivative wahan undefined hai, isliye

[- 1, + 1]

mein koi bhi value legal subgradient hai. Hum

0

isliye pick karte hain kyunki zero par baithe weight ka fate akela data gradient decide karta hai: nonzero subgradient weight ko zero se off nudge karta rehta aur sparsity destroy kar deta, jabki

0

ek genuinely useless weight ko zero par rest karne deta hai. Yahi deliberate choice L1 ko ek sparsity engine banati hai.

p = 0

wala dropout kya reduce ho jaata hai?

Koi bhi neuron kabhi drop nahi hota aur scale factor

\frac{1}{1 - p} = 1

hai, isliye yeh koi dropout ke bina ek ordinary network hai — regularization switch off hai.

p \to 1

wala dropout kya karta hai, aur yeh useless kyun hai?

Almost har neuron har pass mein zero ho jaata hai, isliye koi signal forward flow nahi karta aur scale

\frac{1}{1 - p} \to \infty

kuch survivors ko blow up kar deta hai. Learning stall ho jaati hai — network ke paas essentially koi active capacity nahi hai.

Agar ek feature column exactly duplicate ho, toh L1 aur L2 us pair ko alag kaise treat karte hain?

L2 weight ko roughly evenly dono copies mein split karta hai (use koi bhi single large weight pasand nahi). L1 tend karta hai ek copy rakhne aur doosri zero karne ka, kyunki ek par weight concentrate karna same L1 penalty cost karta hai par yeh ek corner solution hai.

Recall Quick self-test

Ek-line L1-vs-L2 discriminator ::: L1 ek constant subtract karta hai → exact zeros → sparsity; L2 $(1 - η λ)$ se multiply karta hai → proportional shrink → smooth small weights. Woh ek dropout rule jo log galat karte hain ::: Inverted dropout ke saath tum kept activations ko $1/ (1 - p)$ se training ke dauran scale karte ho aur test time par kuch nahi karte; classic dropout isko $(1 - p)$ se test time par scale karta hai.