Hyperparameter tuning for deep nets — Hinglish notes

3.2.15 · AI-ML › Training Deep Networks

Intuition Badi picture

Ek neural net mein do tarah ke knobs hote hain. Parameters (weights $W$ , biases $b$ ) gradient descent se seekhe jaate hain. Hyperparameters woh knobs hain jo aap training se pehle set karte ho, jo control karte hain ki learning kaise hogi — learning rate, batch size, depth, width, regularization, wagera. Gradient descent inhe tune nahi kar sakta kyunki training loss inke w.r.t. kisi usable tarike se differentiable nahi hai (jaise "number of layers" discrete hai, aur learning rate loss ke bahar kaam karta hai). To hum inhe ek alag outer loop se dhundhte hain jo validation set se guide hota hai.

YEH KYUN MATTER KARTA HAI: same architecture sirf better hyperparameters se 60% se 95% accuracy tak ja sakti hai. 80/20 sach: learning rate sabse zyada dominate karta hai. Pehle isko sahi karo; baaki sab fine-tuning hai.

1. Hyperparameter exactly kya hota hai?

DO nested minimizations kyun? Agar hum $λ$ ko training loss pe tune karte, to hum bas wohi pick karte jo data ko memorize kare (jaise zero regularization, bahut bada model). Humein generalization detect karne ke liye ek held-out signal chahiye. Yahi validation set ka poora kaam hai.

2. Main hyperparameters (impact ke hisaab se ranked)

Rank	Hyperparameter	Typical search range	Search kahan...
1	Learning rate $η$	$1 0^{- 5}$ → $1 0^{- 1}$	log scale
2	LR schedule / warmup	step, cosine, warmup steps	—
3	Batch size $B$	16 → 4096	powers of 2
4	Regularization: L2 $λ_{L 2}$ , dropout $p$	$1 0^{- 6}$ – $1 0^{- 2}$ ; $0$ – $0.5$	log / linear
5	Architecture: depth $D$ , width $H$	small→large	discrete
6	Optimizer & momentum $β_{1}, β_{2}$	Adam defaults $0.9, 0.999$	linear-in- $(1 - β)$

Figure — Hyperparameter tuning for deep nets

3. LOG scale pe kyun search karein? (Derivation from scratch)

WHAT chahiye: samples jo orders of magnitude mein evenly spread hon, raw values mein nahi.

WHY: $η$ ka effect roughly multiplicative hota hai. $0.001 \to 0.002$ (×2) jaana training utna hi change karta hai jitna $0.01 \to 0.02$ (×2), chahe absolute gaps ( $0.001$ vs $0.01$ ) 10× alag hon. Agar hum $[1 0^{- 5}, 1 0^{- 1}]$ mein uniformly sample karein, ~90% samples $[1 0^{- 2}, 1 0^{- 1}]$ mein land karenge aur hum chhoti values almost kabhi test nahi karenge.

HOW — sampling rule derive karo. Exponent ko uniformly sample karo: $u \sim U (a, b), η = 1 0^{u} .$ Phir density ke liye, $u = lo g_{10} η$ ke saath, $\frac{d u}{d η} = \frac{1}{η l n 10}$ , to $p (η) = p (u) \frac{d u}{d η} = \frac{1}{b - a} \cdot \frac{1}{η l n 10} \propto \frac{1}{η} .$

4. Search strategies

5. Worked example: learning rate dhundhna

6. Regularization hyperparameters

7. Ek practical recipe (80/20)

Sanity: ek tiny batch (10 samples) ko ~0 loss tak overfit karo → confirm karta hai ki code kaam karta hai.
$η$ dhundho LR range test se (Section 5). Yeh akele aapko 80% wahan pahuncha deta hai.
Schedule fix karo: warmup + cosine decay ek strong default hai.
Random search $η$ , weight decay, dropout ek saath (LR & decay ke liye log scale).
Early stopping / ASHA: jinki val loss kuch epochs ke baad lag kare unhe khatam karo.
Tabhi architecture tune karo. Test set ko end mein sirf ek baar touch karo.

Flashcards

Hyperparameter ko parameter se kya alag karta hai?

Parameters (

W, b

) training loss pe gradient descent se seekhe jaate hain; hyperparameters training se pehle set kiye jaate hain aur control karte hain ki learning kaise hogi, validation set pe tune kiye jaate hain.

Gradient descent learning rate ko directly kyun tune nahi kar sakta?

Training loss iske w.r.t. usefully differentiable nahi hai (LR loss ke bahar kaam karta hai; kaafi hypers discrete hain), aur hypers pe training loss minimize karna bas wohi pick karega jo overfit kare — humein held-out validation signal chahiye.

Learning rate log scale pe sample kyun karein?

Iska effect multiplicative hai; log-uniform sampling har decade ko equal probability deta hai.

η = 1 0^{u}

u \sim U (a, b)

sample karne se density

p (η) \propto 1/ η

milti hai.

Kaunsa single hyperparameter sabse zyada impact karta hai aur kyun?

Learning rate — yeh har gradient step ko multiply karta hai, control karta hai ki training converge, diverge, ya crawl kare.

Random search grid search se better kyun hai?

Sirf kuch hypers matter karte hain; same budget ke liye, random search important dimension ko bahut zyada distinct values pe test karta hai jabki grid unimportant repeats pe trials waste karta hai.

Batch size ke liye linear scaling rule batao.

Batch size ko

k

se multiply karo ⇒ learning rate ko

k

se multiply karo (bade

k

ke liye warmup add karo), taaki har example ke liye expected parameter displacement roughly constant rahe.

Weight decay ko "decay" kyun kehte hain?

L2 update ban jaata hai

θ \leftarrow (1 - η λ) θ - η g

; factor

(1 - η λ) < 1

weights ko har step mein 0 ki taraf shrink karta hai.

Test set sirf ek baar kyun use karna chahiye?

Test set use karke ki gayi koi bhi tuning model mein info leak karti hai, ise overfit karti hai aur reported score ko optimistically bias karti hai.

LR range test kya hai?

Har batch mein

η

ko geometrically badhao, loss vs

lo g η

plot karo; steepest descent ke paas

η

pick karo, ~ek order neeche wahan se jahan loss diverge hota hai.

Recall Feynman: 12-saal ke bache ko samjhao

Socho tum cakes bake kar rahe ho. Recipe ki maatraaein (aata, cheeni) weights ki tarah hain — oven inhe "seekhta" hai jaise jaise bake hota hai. Lekin oven ka temperature, baking time, aur cake ka size woh cheezein hain jo tum baking se pehle decide karte ho — yeh hyperparameters hain. Temperature bahut zyada karo aur cake jal jaata hai (loss explode); bahut kam karo aur yeh hamesha kaccha rehta hai (bahut dheere seekhta hai). Aap oven ko apna temperature khud nahi choose karne de sakte usi cake ka swaad lekar jo woh bake kar raha hai (yeh training set hai) — tumhe ek doosra tester chahiye (validation) jo bataye ki temperature achha tha. Aur tum ek final judge (test set) rakhte ho jo sirf ek baar taste karta hai taaki tum unke liye adjust karke cheat na kar sako.

Connections

Gradient Descent and SGD — inner loop jise hyperparameters modulate karte hain.
Learning Rate Schedules and Warmup — cosine, step decay.
Regularization - L2 and Dropout — capacity-control hyperparameters.
Bias-Variance Tradeoff — validation loss selection signal hai.
Batch Normalization — LR aur batch size ke saath interact karta hai.
Cross-Validation and Data Splits — train/val/test discipline.
Adam and Momentum Optimizers — $β_{1}, β_{2}$ hyperparameters ke roop mein.