Exercises — Hyperparameter tuning for deep nets

A parameter is anything gradient descent updates by minimizing the training loss; a hyperparameter is a knob you fix before that loop and tune on validation.

• (a) weight $W_{ij}$ → parameter
• (b) $\eta$ → hyperparameter (it acts outside the loss; the loss isn't usefully differentiable in it)
• (c) $D$ → hyperparameter (and discrete — you can't take a gradient in "3.5 layers")
• (d) bias $b$ → parameter
• (e) dropout $p$ → hyperparameter

Because the update is $\theta \leftarrow \theta -\eta g$: the factor $\eta$ multiplies every gradient of every parameter at every step, so it alone decides whether training converges, crawls, or diverges. Every other knob is a refinement that only matters once $\eta$ is roughly right.

WHAT: we sample the exponent uniformly, then raise 10 to it — that is what makes decades equally likely (see the figure below). The exponent range is $[a,b]=[-5,-1]$, width $b-a=4$. Map $u\in [0,1]$ into it: $\text{exp}=a+u(b-a)=-5+0.25\times 4=-5+1=-4$. $\eta =10^{-4}.$

WHY this rule: doubling the batch halves the gradient noise but also means one big step now covers what several small steps used to; to keep the expected parameter displacement the same, scale $\eta$ by the same factor $k$ as the batch. Factor $k=512/128=4$. ${\eta }^{\prime }=k\eta =4\times 0.01=\mathrm{0.04.}$ (In practice you'd add warmup, because the linear rule frays for very large $k$.)

The decay factor is $1-\eta \lambda =1-(0.1)(0.05)=1-0.005=0.995$, so it shrinks by $0.5\%$ each step. With $g=0$, ${\theta }_2=(0.995{)}^2{\theta }_0=0.990025\times 1=0.990025$.

Grid: the important knob only takes its 5 grid values; the other two dimensions just repeat those 5 values 25 times each — so you get 5 distinct settings of what matters and 120 wasted duplicates. Random: each of the 125 runs draws an independent value of the important knob, so you test it at 125 distinct values. One sentence: grid search wastes budget re-testing along dimensions that don't matter, while random search automatically spends its whole budget resolving whichever dimension does matter.

WHAT the three regions mean: flat = $\eta$ too small (negligible progress); steep = the productive band; rising = $\eta$ too large, updates overshoot and the loss diverges. The minimum-loss point on this curve sits right next to the divergence region — training there is on the edge of instability, and a slightly noisier batch can tip it into blow-up. Choosing roughly one decade below ($3\times 10^{-3}$) gives a margin of safety: still deep in the productive band, but far from the cliff.

(a) Grid: $5^5=3125$ trainings $\times 2\text{h}=6250$ GPU-hours. Budget is 240 h → grid needs $\approx 26\times$ your budget → infeasible. (b) Random full-length trials: $240/2=120$ trials. (c) With ASHA, average cost per started trial $=\frac{3}{4}(0.5)+\frac{1}{4}(2)=0.375+0.5=0.875\text{h}$. Trials you can start $=240/0.875\approx 274$ → about 274 trials (vs 120), because most die cheaply. Early stopping lets you sample far more configurations for the same budget.

Step 1 — draw $\eta$. Range $[a,b]=[-4,-1]$, width $3$. Exponent $=-4+0.5\times 3=-2.5$, so ${\eta }_{\text{drawn}}=10^{-2.5}\approx 0.0031623$. Step 2 — scale for batch. $k=512/128=4$, so ${\eta }_{\text{final}}=4\times 10^{-2.5}=4\times 0.0031623\approx 0.012649$. The two rules compose cleanly: log-uniform decides the order of magnitude, linear scaling corrects for the batch size on top of it.

A defensible order (matches the parent's 80/20 recipe), each step earning its place:

1. Overfit a tiny batch (10 samples) to ~0 loss. Protects against: silent bugs — if the net can't memorize 10 points, the code/architecture is broken; tune nothing yet.
2. LR range test to find $\eta$. Protects against: wasting the whole budget with a bad learning rate. This is the highest-impact single move ($\eta$ is rank #1).
3. Pick a schedule (warmup + cosine decay). Protects against: instability early (warmup) and slow crawling late (decay); see Learning Rate Schedules and Warmup.
4. Random search $\eta$, ${\lambda }_{L2}$, dropout together — log scale for $\eta$ and ${\lambda }_{L2}$. Protects against: grid's combinatorial explosion and against poor resolution on the important axis.
5. Early stopping / ASHA. Protects against: spending full budget on doomed runs; buys breadth (L4.1).
6. Tune architecture (depth/width) last, as a validation-scored hyperparameter. Protects against: overfitting from blindly maxing capacity (bigger ≠ better without regularization).
7. Touch the test set once. Protects against: optimistic bias — any choice made using the test set leaks into the model. See Cross-Validation and Data Splits.

WHY change of variables: $u$ is flat, but we care about the spread of $\eta$; to move a density from $u$ to $\eta$ we multiply by $\left|\frac{du}{d\eta }\right|$ (the change-of-variable rule for probability densities — it keeps total probability at 1). Since $u={\log }_{10}\eta =\frac{\ln \eta }{\ln 10}$, $\frac{du}{d\eta }=\frac{1}{\eta \ln 10}.$ The density of $u$ is the flat $p(u)=\frac{1}{b-a}$ on $[a,b]$. Therefore