Visual walkthrough — Hyperparameter tuning for deep nets

This is the visual companion to the parent topic. If you have never met the word learning rate, start at Step 1 — we assume nothing.

Step 1 — What is the learning rate, as a picture?

WHAT: picture a valley (the loss). You are a ball on its side wanting to reach the bottom.

WHY $\eta$ matters so much: it multiplies every step, forever. Too small and you crawl; too big and you leap clean over the bottom and up the far wall. Look at the figure: three balls, three values of $\eta$.

Step 2 — The knob is multiplicative, not additive

WHAT: compare two changes that look very different on a ruler but feel the same to the network.

WHY this is the whole story: if the effect lives in ratios, then the sensible values of $\eta$ are spread across orders of magnitude — $10^{-5},10^{-4},\dots ,10^{-1}$ — not evenly along a normal number line. A knob whose effect is multiplicative should be searched multiplicatively.

PICTURE: the figure shows a normal ("linear") ruler on top and a "×10 each tick" ruler (a log ruler) below. The four candidate learning rates crowd into one corner on the linear ruler but sit evenly spaced on the log ruler.

Step 3 — The disaster of uniform sampling (the thing we must avoid)

WHAT: draw many random $\eta$ uniformly and see where they land.

WHY it fails: the range $[10^{-2},10^{-1}]$ is $0.09$ wide, while $[10^{-3},10^{-2}]$ is only $0.009$ wide — ten times narrower — and $[10^{-5},10^{-4}]$ is a hundred times narrower still. Uniform sampling drops points by area, so roughly 90% of all draws fall in that top decade $[10^{-2},10^{-1}]$, and the small values almost never get tested.

PICTURE: the histogram below piles nearly all samples into the largest decade. The tiny learning rates — often exactly the ones a delicate network needs — are essentially never explored.

Step 4 — The fix: sample the exponent, not the value

WHAT: roll a uniform number for the power of ten, then take that power of ten.

WHY it works: picking $u$ evenly on $[-5,-1]$ means each decade — $[-5,-4]$, $[-4,-3]$, … — is an equal-width slice of $u$, so each gets an equal share of samples. Convert back and every ×10 band of $\eta$ is explored equally.

PICTURE: the same random draws as Step 3, but now generated by $\eta =10^u$. On the log ruler they are spread flat and even; every decade has its fair handful of points.

Step 5 — Proving it: the density comes out as $1/\eta$

WHAT we do: transform the flat density $p(u)=\frac{1}{b-a}$ from $u$-world into $\eta$-world using that change-of-variables rule $p(\eta )=p(u)\left|\frac{du}{d\eta }\right|.$

• $p(u)=\frac{1}{b-a}$ = flat: uniform in $u$ means constant height.
• $\left|\frac{du}{d\eta }\right|$ = the stretch factor: how many units of $u$ correspond to one unit of $\eta$ right here.

WHY the derivative of the log: since $u={\log }_{10}\eta$, we need $\frac{du}{d\eta }$. Using ${\log }_{10}\eta =\frac{\ln \eta }{\ln 10}$ and $\frac{d}{d\eta }\ln \eta =\frac{1}{\eta }$: $\frac{du}{d\eta }=\frac{1}{\eta \ln 10}.$

Put it together (and note $\eta >0$, so this is already positive — the $|\cdot |$ is safe): $p(\eta )=\frac{1}{b-a}\cdot \frac{1}{\eta \ln 10}\propto \frac{1}{\eta }.$

• $\frac{1}{b-a}$ and $\frac{1}{\ln 10}$ = fixed constants, they don't depend on $\eta$.
• $\frac{1}{\eta }$ = the shape that matters: density falls off like one-over-$\eta$.

WHY $1/\eta$ is exactly "equal per decade": the chance of landing in a band $[{\eta }_1,{\eta }_2]$ is the area under $\frac{1}{\eta }$, which is $\ln {\eta }_2-\ln {\eta }_1$ — it depends only on the ratio ${\eta }_2/{\eta }_1$. Every decade has ratio $10$, so every decade has equal probability. ∎

PICTURE: the smooth $1/\eta$ curve, with three decades shaded — each shaded area is equal, even though the curve is far taller on the left.

Step 6 — Edge cases: zero, negatives, and endpoints

Case $\eta \to 0$. On a log scale $0$ is infinitely far to the left (${\log }_{10}0=-\infty$), so it is never sampled and never should be — a zero learning rate means no learning at all (the update $-\eta g$ becomes $0$). This is a feature: log sampling automatically refuses the useless value.

Case $\eta <0$. A negative learning rate would step uphill, increasing the loss — nonsense. Log sampling cannot produce it, since $10^u>0$ for every real $u$. Safe by construction.

Case: the endpoints $a,b$. These are your honest bounds. Choose $a$ small enough that the loss is still flat (nothing happens) and $b$ large enough that training diverges, so the useful region sits inside the range, never clipped at an edge.

Case: a knob that lives in $[0,1)$, like dropout probability $p$ or momentum $\beta$. Here the interesting action is near the top edge ($\beta \to 1$). Log-sampling $\beta$ itself is wrong; instead log-sample the distance from 1, i.e. $1-\beta$. That is why the parent's table says "linear-in-$(1-\beta )$".

PICTURE: a number line marking the forbidden zone ($\eta \le 0$, greyed), the flat zone, the productive band, and the divergence zone — showing where good bounds $a,b$ should sit.

Step 7 — Random beats grid, because of this scale

WHAT: with a fixed budget of runs, compare grid (fixed lattice of values) against random (each run gets fresh random values) when only one of two knobs actually matters.

WHY random wins: on a grid, the important knob is tested at only as many distinct values as one grid axis (say 5), and whole rows are wasted repeating it. Random search gives the important knob a different value on every single run, so 25 runs test it at 25 distinct spots — and because we draw those on the log scale, they're spread across all decades, not bunched up.

PICTURE: the classic Bergstra–Bengio panel — grid projects onto 5 columns; random projects onto 25 spread-out points along the knob that matters.

See Cross-Validation and Data Splits for which data measures each run's score, and Learning Rate Schedules and Warmup for what to do after you've found a good $\eta$.

The one-picture summary

The whole derivation compressed: a multiplicative knob (top) → uniform-in-value wastes 90% on one decade (left) → sample the exponent instead (middle) → density becomes $1/\eta$, equal per decade, never negative, never zero (right).

Quick self-check