Hyperparameter tuning for deep nets

1. What exactly is a hyperparameter?

WHY two nested minimizations? If we tuned $\lambda$ on the training loss, we'd just pick whatever memorizes the data (e.g. zero regularization, huge model). We need a held-out signal to detect generalization. That is the validation set's whole job.

2. The main hyperparameters (ranked by impact)

Rank	Hyperparameter	Typical search range	Search on...
1	Learning rate $\eta$	$10^{-5}$ → $10^{-1}$	log scale
2	LR schedule / warmup	step, cosine, warmup steps	—
3	Batch size $B$	16 → 4096	powers of 2
4	Regularization: L2 $\lambda_{L2}$ , dropout $p$	$10^{-6}$ – $10^{-2}$ ; $0$ – $0.5$	log / linear
5	Architecture: depth $D$ , width $H$	small→large	discrete
6	Optimizer & momentum $\beta_1,\beta_2$	Adam defaults $0.9, 0.999$	linear-in- $(1-\beta)$

Figure — Hyperparameter tuning for deep nets

3. Why search on a LOG scale? (Derivation-from-scratch)

WHAT we want: samples spread evenly across orders of magnitude, not across raw values.

WHY: the effect of $\eta$ is roughly multiplicative. Going $0.001 \to 0.002$ (×2) changes training as much as $0.01 \to 0.02$ (×2), even though the absolute gaps ( $0.001$ vs $0.01$ ) differ 10×. If we sampled uniformly in $[10^{-5},10^{-1}]$ , ~90% of samples would land in $[10^{-2},10^{-1}]$ and we'd almost never test the small values.

HOW — derive the sampling rule. Sample the exponent uniformly: $u \sim \mathcal U(a,b),\qquad \eta = 10^{u}.$ Then for the density, with $u=\log_{10}\eta$ , $\frac{du}{d\eta}=\frac{1}{\eta\ln 10}$ , so $p(\eta) = p(u)\left|\frac{du}{d\eta}\right| = \frac{1}{b-a}\cdot\frac{1}{\eta\ln 10}\ \propto\ \frac{1}{\eta}.$

4. Search strategies

5. Worked example: finding a learning rate

6. Regularization hyperparameters

7. A practical recipe (80/20)

Sanity: overfit a tiny batch (10 samples) to ~0 loss → confirms code works.
Find $\eta$ with an LR range test (Section 5). This alone gets you 80% of the way.
Fix a schedule: warmup + cosine decay is a strong default.
Random search $\eta$ , weight decay, dropout together (log scale for LR & decay).
Early stopping / ASHA: kill runs whose val loss lags after a few epochs.
Only then tune architecture. Touch the test set once at the end.

Flashcards

What distinguishes a hyperparameter from a parameter?

Parameters (

W,b

) are learned by gradient descent on training loss; hyperparameters are set before training and control how learning happens, tuned on the validation set.

Why can't gradient descent tune the learning rate directly?

The training loss isn't usefully differentiable w.r.t. it (LR acts outside the loss; many hypers are discrete), and minimizing training loss over hypers would just pick whatever overfits — we need the held-out validation signal.

Why sample learning rate on a log scale?

Its effect is multiplicative; log-uniform sampling gives each decade equal probability. Sampling

\eta=10^u

u\sim U(a,b)

yields density

p(\eta)\propto 1/\eta

Which single hyperparameter has the largest impact and why?

Learning rate — it multiplies every gradient step, controlling whether training converges, diverges, or crawls.

Why does random search beat grid search?

Only a few hypers matter; for the same budget, random search tests the important dimension at many distinct values while grid wastes trials on unimportant repeats.

State the linear scaling rule for batch size.

Multiply batch size by

k

⇒ multiply learning rate by

k

(add warmup for large

k

), to keep the expected parameter displacement per example roughly constant.

Why is weight decay called "decay"?

The L2 update becomes

\theta\leftarrow(1-\eta\lambda)\theta-\eta g

; the factor

(1-\eta\lambda)<1

shrinks weights toward 0 each step.

Why must the test set be used only once?

Any tuning done using the test set leaks info into the model, overfitting it and biasing the reported score optimistically.

What is the LR range test?

Increase

\eta

geometrically each batch, plot loss vs

\log\eta

; pick

\eta

near steepest descent, ~one order below where loss diverges.

Recall Feynman: explain to a 12-year-old

Imagine baking cakes. The recipe amounts (flour, sugar) are like weights — the oven figures those out as it "learns" to bake. But oven temperature, baking time, and cake size are things you decide before baking — those are hyperparameters. Set the temperature too high and the cake burns (loss explodes); too low and it's raw forever (learns too slowly). You can't let the oven pick its own temperature by tasting the same cake it's baking (that's the training set) — you need a second tester (validation) to tell you the temperature was good. And you keep one final judge (test set) who tastes only once so you can't cheat by adjusting for them.

Connections

Gradient Descent and SGD — the inner loop hyperparameters modulate.
Learning Rate Schedules and Warmup — cosine, step decay.
Regularization - L2 and Dropout — capacity-control hyperparameters.
Bias-Variance Tradeoff — validation loss is the selection signal.
Batch Normalization — interacts with LR and batch size.
Cross-Validation and Data Splits — train/val/test discipline.
Adam and Momentum Optimizers — $\beta_1,\beta_2$ as hyperparameters.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, ek neural net mein do tarah ke knobs hote hain. Parameters (weights aur biases) model khud gradient descent se seekh leta hai. Lekin hyperparameters — jaise learning rate, batch size, number of layers, dropout — ye tumhe training se pehle set karne padte hain. Gradient descent inhe tune nahi kar sakta, isliye ek alag "outer loop" chahiye jo validation set ke score dekh ke best value choose kare. Yaad rakho: training set se params fit karo, validation set se hyperparameters choose karo, aur test set ko sirf ekdum end mein ek baar chhoona — warna tum test pe overfit ho jaoge aur accuracy jhoothi lagegi.

Sabse important knob hai learning rate ( $\eta$ ). Ye har gradient step ko multiply karta hai, isliye agar bahut bada rakhoge to loss udd jayega (NaN), aur bahut chhota rakhoge to training kachhue ki speed se chalegi. 80/20 rule: pehle sirf learning rate theek karo, baaki sab baad mein. Ek smart trick hai LR range test — $\eta$ ko chhote value se dhire-dhire badhao har batch mein, aur loss vs $\log\eta$ ka graph banao; jaha loss sabse tezi se gir raha hai uske paas, divergence point se ek order neeche, wahi accha $\eta$ hai.

Ek aur important cheez: learning rate ko hamesha log scale pe search karo, kyunki uska effect multiplicative hota hai — $0.001\to0.002$ utna hi farak laata hai jitna $0.01\to0.02$ . Aur random search grid search se better hai, kyunki sirf kuchh hi hyperparameters actually matter karte hain; random search apna budget automatically important dimension pe kharch kar deta hai. Weight decay ( $1-\eta\lambda$ factor) weights ko chhota rakhta hai taaki overfitting na ho. Bas yaad rakho recipe: overfit tiny batch → LR find karo → schedule → batch size → regularization → architecture.

1. What exactly is a hyperparameter?

2. The main hyperparameters (ranked by impact)

3. Why search on a LOG scale? (Derivation-from-scratch)

4. Search strategies

5. Worked example: finding a learning rate

6. Regularization hyperparameters

7. A practical recipe (80/20)

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Training Deep Networks

Connections