Hyperparameter tuning for deep nets: Worked examples

The scenario matrix

Every worked example below is tagged with a cell ID from this table. Together they cover every cell.

Cell	Situation class	Degenerate / edge sub-case
A	Learning rate too small	$η \to 0$ : no progress
B	Learning rate too large	crosses the divergence edge
B2	Learning rate at the exact edge	$η c = 2$ : bounces forever
C	Learning rate just right	the productive band $0 < η c < 1$ AND $η c = 1$
D	Log-scale sampling of $η$	equal mass per decade
E	Batch-size ↔ LR coupling	linear scaling rule, $k$ factor
F	Weight decay shrink factor	$λ = 0$ (no decay), large $λ$ (underfit), unstable $η λ \geq 2$
G	Grid vs random search budget	only 1 of many knobs matters
H	Real-world word problem	wall-clock time for a search
I	Exam twist	"tuned on the test set" trap

To reason about cells A–C we need one honest, solvable toy model of a single gradient-descent step. Let us build it first, because every LR example rides on it.

Figure — Hyperparameter tuning for deep nets

Recall The convergence rule you must remember

Since $θ_{t} = (1 - η c)^{t} θ_{0}$ , the weight goes to $0$ (the minimum) only if $∣1 - η c ∣ < 1$ , i.e. $0 < η < \frac{2}{c}$ .

$0 < η c < 1$ : $r \in (0, 1)$ — smooth shrink, no overshoot (Cell C, non-boundary).
$η c = 1$ : $r = 0$ — one perfect jump to the minimum (Cell C, boundary).
$1 < η c < 2$ : $r \in (- 1, 0)$ — overshoots but still converges (oscillates in) (Cell B).
$η c = 2$ : $r = - 1$ — bounces forever, never converging (Cell B2).
$η c > 2$ : $∣ r ∣ > 1$ — magnitude grows without bound, $∣ θ_{t} ∣ \to \infty$ (Cell B).

$η$ barely moves Loss $L = \frac{1}{2} c θ^{2}$ with curvature $c = 1$ , start $θ_{0} = 1$ , learning rate $η = 0.01$ . How much of the way to the minimum ( $θ = 0$ ) have you travelled after 1 step? After 100 steps?

Forecast: guess the fraction of the gap closed after 1 step, and whether 100 steps is "basically done".

Compute the shrink factor $r = 1 - η c = 1 - 0.01 \cdot 1 = 0.99$ . Why this step? $r$ is the multiplier from the lab-rat model; it is all we need.
After 1 step $θ_{1} = r θ_{0} = 0.99$ . Gap closed $= 1 - 0.99 = 0.01 = 1%$ . Why this step? "Fraction closed" $= 1 - r$ directly measures progress per step.
After 100 steps $θ_{100} = r^{100} = 0.9 9^{100} = 0.3660$ . Gap closed $= 63.4%$ . Why this step? Steps compound multiplicatively, so we raise $r$ to the power $t$ .

Verify: using the natural log, $ln (0.99) \approx - 0.01$ , so $0.9 9^{100} = e^{100 l n 0.99} \approx e^{- 1} = 0.368$ . Matches $0.3660$ . So after one hundred steps you are barely two-thirds there — the classic "loss creeps down painfully slowly" symptom. Units: $θ$ dimensionless, factor dimensionless. ✓

Cell C — learning rate just right (both sub-cases)

Worked example C-i: the productive band

$0 < η c < 1$ (smooth convergence, no overshoot) Same loss, $c = 1$ , $θ_{0} = 1$ . Try a healthy $η = 0.5$ . Where are you after 1, 2, 3 steps, and how many steps to close 99% of the gap?

Forecast: guess whether it overshoots (sign flips) or slides in smoothly, and roughly how many steps to be "basically done".

Shrink factor $r = 1 - η c = 1 - 0.5 = 0.5$ , which lies in $(0, 1)$ . Why this step? $r \in (0, 1)$ is exactly the "smooth shrink, no overshoot" regime — the genuinely healthy learning rate, distinct from Cell A's crawling $r = 0.99$ .
Iterate: $θ_{1} = 0.5, θ_{2} = 0.25, θ_{3} = 0.125$ — all positive, monotonically shrinking, never crossing zero. Why this step? positive $r < 1$ means each step lands between you and the minimum, never past it — the opposite of Cell B's sign-flipping overshoot.
Steps to close 99%: solve $0. 5^{t} \leq 0.01 \Rightarrow t \geq \frac{ln 0.01}{ln 0.5} = \frac{- 4.605}{- 0.693} = 6.64$ , so $t = 7$ steps. Why this step? compare with Cell A which needed hundreds of steps — this is what "just right" buys you: order-of-magnitude faster convergence with no instability.

Verify: $0. 5^{7} = 0.0078 \leq 0.01$ ✓ while $0. 5^{6} = 0.0156 > 0.01$ , so $t = 7$ is the first step past 99% ✓. Monotone positive sequence confirms no overshoot ✓.

Cell B — learning rate too large (crossing the edge)

Worked example Overshoot and divergence

Same loss, $c = 1$ , $θ_{0} = 1$ . Compare $η = 1.5$ and $η = 2.5$ over a few steps.

Forecast: which one converges (oscillating inward) and which one explodes?

For $η = 1.5$ : $r = 1 - 1.5 = - 0.5$ . $∣ r ∣ = 0.5 < 1$ → converges but sign flips each step. Sequence: $1, - 0.5, 0.25, - 0.125, \dots$ Why this step? Negative $r$ means you jump past the minimum, landing on the other side.
For $η = 2.5$ : $r = 1 - 2.5 = - 1.5$ . $∣ r ∣ = 1.5 > 1$ → magnitude grows every step, so $∣ θ_{t} ∣ \to \infty$ (it diverges mathematically; in floating point it later overflows, but the true behaviour is unbounded growth, not literally "NaN"). Sequence: $1, - 1.5, 2.25, - 3.375, \dots \to \pm \infty$ . Why this step? $∣ r ∣ > 1$ multiplies the error up every step — this is the "loss shoots up, diverges" case in the range test.

Verify: The boundary is $η c = 2$ , i.e. $η = 2$ . Indeed $1.5 < 2 < 2.5$ splits the two behaviours exactly as predicted. See figure s02 below. ✓

Cell B2 — the exact edge $η c = 2$

Worked example The perpetual bounce

Same loss, $c = 1$ , $θ_{0} = 1$ . Set $η = 2$ exactly (so $η c = 2$ ). What happens after 1, 2, 3, 4 steps? Does it ever converge?

Forecast: guess whether it shrinks, grows, or does something in between.

Shrink factor $r = 1 - η c = 1 - 2 = - 1$ . Why this step? This is the exact boundary of the stability rule $∣ r ∣ < 1$ ; here $∣ r ∣ = 1$ .
Iterate: $θ_{1} = - 1, θ_{2} = + 1, θ_{3} = - 1, θ_{4} = + 1, \dots$ Why this step? multiplying by $- 1$ just flips the sign; magnitude never changes.
It never converges and never diverges — it oscillates forever at magnitude $1$ . Why this step? $∣ r ∣ = 1$ neither shrinks ( $∣ r ∣ < 1$ ) nor grows ( $∣ r ∣ > 1$ ) the weight.

Verify: $∣ θ_{t} ∣ = ∣ r ∣^{t} ∣ θ_{0} ∣ = 1^{t} \cdot 1 = 1$ for all $t$ — constant magnitude, so the loss $L = \frac{1}{2} \cdot 1 \cdot θ_{t}^{2} = \frac{1}{2}$ is frozen forever, never reaching $0$ . This is exactly the " $η c = 2$ , $r = - 1$ , bounces forever" line of the convergence rule. ✓

Cell D — log-scale sampling

Worked example Do log-uniform samples really spread evenly across decades?

You sample $η = 1 0^{u}$ with $u \sim U (- 5, - 1)$ (uniform between $- 5$ and $- 1$ ). What is the probability a sample lands in the decade $[1 0^{- 3}, 1 0^{- 2}]$ ? In $[1 0^{- 2}, 1 0^{- 1}]$ ?

Forecast: guess whether the tiny-value decade and the big-value decade get equal chance.

In $u$ -space, $[1 0^{- 3}, 1 0^{- 2}]$ corresponds to $u \in [- 3, - 2]$ , width $1$ . Why this step? Taking $lo g_{10}$ turns "a decade" into "an interval of length 1 in $u$ " (here $u = lo g_{10} η$ , base-10 exactly because decades are powers of 10).
Total $u$ -range is $[- 5, - 1]$ , width $4$ . Probability of any width-1 sub-interval $= \frac{1}{4} = 0.25$ . Why this step? Uniform $u$ gives probability = interval length ÷ total length.
$[1 0^{- 2}, 1 0^{- 1}]$ is $u \in [- 2, - 1]$ , also width 1 → probability $0.25$ too. Why this step? Every decade is width 1 in $u$ , so all four decades get $0.25$ .

Verify: four decades × $0.25 = 1.0$ , a valid distribution. Compare to uniform in $η$ : the total width of $[1 0^{- 5}, 1 0^{- 1}]$ is $1 0^{- 1} - 1 0^{- 5} = 0.09999$ (that is $0.1 - 0.00001 = 0.09999$ exactly — the trailing digits are literally $0.09999$ , not a rounded $0.1$ ; keep all five digits so the arithmetic below is honest). Then $P ([1 0^{- 3}, 1 0^{- 2}]) = \frac{1 0 ^{- 2} - 1 0 ^{- 3}}{0.09999} = \frac{0.009}{0.09999} \approx 0.0900$ , while $P ([1 0^{- 2}, 1 0^{- 1}]) = \frac{0.09}{0.09999} \approx 0.9001$ — a ~10× bias to the big decade, exactly the failure the parent warned about. ✓

Cell E — batch-size ↔ learning-rate coupling

Worked example Linear scaling rule

You trained well with batch size $B = 256$ and learning rate $η = 0.1$ . You now move to $B = 1024$ to use a bigger GPU. What learning rate should you start with?

Forecast: guess before computing — bigger, smaller, or same $η$ ?

Let $\overset{g}{ˉ}$ denote the average gradient over a batch — literally the mean of the per-example gradients $g_{1}, \dots, g_{B}$ , i.e. $\overset{g}{ˉ} = \frac{1}{B} \sum_{i} g_{i}$ . It is the direction SGD actually steps in.

Scaling factor $k = B_{new} / B_{old} = 1024/256 = 4$ . Why this step? The linear scaling rule matches expected parameter displacement; the factor that changed is exactly $k$ .
New learning rate $η^{'} = k η = 4 \times 0.1 = 0.4$ . Why this step? One big step over $k$ × more examples must move as far as $k$ small steps did, so $η$ scales by $k$ (see parent §5 derivation).

Verify: Check the displacement match using $\overset{g}{ˉ}$ (the average gradient). $k$ small steps move $\approx η k \overset{g}{ˉ} = 0.1 \times 4 \times \overset{g}{ˉ} = 0.4 \overset{g}{ˉ}$ . One big step moves $η^{'} \overset{g}{ˉ} = 0.4 \overset{g}{ˉ}$ . Equal ✓. Caveat: for very large $k$ curvature dominates and you need warmup — see Learning Rate Schedules and Warmup. Also relates to noise reduction in Adam and Momentum Optimizers.

Cell F — weight decay shrink factor (all regimes)

Worked example How fast do idle weights decay — and when does decay itself go unstable?

With L2 weight decay the update is $θ \leftarrow (1 - η λ) θ - η g$ . Take $η = 0.1$ , decay $λ = 0.01$ . If a weight receives no gradient signal ( $g = 0$ ), by what fraction has it shrunk after 10 steps? Then examine the degenerate cases $λ = 0$ , large $λ = 1$ , and the stability limit $η λ \geq 2$ .

Forecast: guess the 10-step shrink for $λ = 0.01$ , then what $λ = 0$ , $λ = 1$ , and $λ = 25$ do.

Per-step factor $f = 1 - η λ = 1 - 0.1 \cdot 0.01 = 0.999$ . Why this step? With $g = 0$ the update is pure multiplication by $f$ — this is why it is called "decay". Note $f$ plays exactly the role $r$ did in the LR model.
After 10 steps $θ_{10} = f^{10} θ_{0} = 0.99 9^{10} θ_{0} = 0.99004 θ_{0}$ . Shrinkage $= 1 - 0.99004 = 0.00996 \approx 1%$ . Why this step? decay compounds, so raise $f$ to the power 10.
Degenerate $λ = 0$ : $f = 1$ , weight never shrinks — no regularization at all. Large $λ = 1$ : $f = 1 - 0.1 \cdot 1 = 0.9$ ; after 10 steps $0. 9^{10} = 0.3487$ → weight crushed to 35% even with no gradient → risk of underfitting. Why this step? covers the two useful ends of cell F.
Stability limit: decay is stable only while $∣ f ∣ < 1$ , i.e. $0 < η λ < 2$ . If $η λ = 2$ then $f = - 1$ (weights flip sign forever, like Cell B2); if $η λ > 2$ then $∣ f ∣ > 1$ and the decay term itself blows the weights up. Example $λ = 25$ : $f = 1 - 0.1 \cdot 25 = - 1.5$ , $∣ f ∣ > 1$ → divergence. Why this step? mirrors the LR stability rule — decay is just another multiplicative factor and obeys the same $∣ f ∣ < 1$ condition.

Verify: $0.99 9^{10} \approx e^{- 0.01} = 0.99005$ ✓; $0. 9^{10} = 0.34868$ ✓; and $f (λ = 25) = 1 - 2.5 = - 1.5$ with $∣ f ∣ > 1$ ✓. The bigger $λ$ pulls weights toward 0 harder, matching Regularization - L2 and Dropout and the Bias-Variance Tradeoff (strong decay → high bias).

Cell G — grid vs random search budget

Worked example Distinct values of the ONE knob that matters

You have 3 hyperparameters but only 1 actually affects accuracy. Budget = 125 runs. Grid search uses $5 \times 5 \times 5$ . How many distinct values of the important knob does each strategy test?

Forecast: guess grid's count vs random's count.

Grid: the important knob takes only its 5 grid values; the other $5 \times 5 = 25$ combinations just repeat each of those values. Why this step? A grid tests each axis at a fixed small set — repeated across the others.
So grid tests the important knob at $5$ distinct values (using all 125 runs).
Random: each of the 125 runs draws a fresh independent value → $125$ distinct values (with probability 1 for continuous ranges). Why this step? Independent sampling never wastes a run on a repeated important value.

Verify: $125/5 = 25$ — grid wastes 25 runs per useful value; random wastes none. This is the Bergstra–Bengio result quoted in the parent. Ratio of resolution $= 125/5 = 25 \times$ better for random. ✓

Cell H — real-world word problem

Worked example Will the search finish before the deadline?

A grid search over 5 hyperparameters, 4 values each, on a cluster that trains one config in 90 minutes, with 8 GPUs running configs in parallel (one config per GPU). How many wall-clock days? If your deadline is 7 days, does it fit?

Forecast: guess days, and yes/no on the deadline.

Total configs $= 4^{5} = 1024$ . Why this step? Grid cost is the product of per-axis counts.
Total compute needed $= 1024 \times 1.5 h = 1536$ GPU·hours (this is machine-work, not wall-clock time yet). Why this step? each config = 90 min = 1.5 h of work on one GPU; summing over all configs gives total GPU·hours of work to be done.
Wall-clock time = total GPU·hours ÷ number of GPUs working in parallel $= 1536 GPU\cdoth /8 GPU = 192$ hours $= 192/24 = 8$ days. Why this step? 8 GPUs chew through the 1536 GPU·hours of work 8× faster in real time; the units cancel: GPU·h ÷ GPU = h.
$8 > 7$ → does not fit the deadline. Switch to random search + ASHA early-stopping. Why this step? connects the arithmetic back to the practical recipe.

Verify: $4^{5} = 1024$ ; $1024 \times 1.5 = 1536$ GPU·h; $1536/8 = 192$ h wall-clock; $192/24 = 8$ days ✓. Unit chain: configs × (h/config on one GPU) = GPU·h; ÷ GPU = h; ÷ 24 = days ✓. See Cross-Validation and Data Splits for why you need the val set inside each of these runs.

Cell I — exam twist

Worked example The "tuned on test" bias

A student runs random search, and for each of $N = 50$ configs measures accuracy on the test set, reporting the best. Their true generalization accuracy is $μ = 0.80$ , and each test estimate is noisy with standard deviation $σ = 0.02$ . Roughly how much does "reporting the max of 50 noisy estimates" inflate the reported number?

Forecast: guess whether the reported number is above or below $0.80$ , and by roughly how much.

The reported value is $max$ of 50 draws from $N (0.80, 0.0 2^{2})$ . Why this step? choosing the config by test score = taking the maximum estimate.
The expected max of $N$ standard normals is well approximated (including the $ln ln N$ correction, not just the crude $2 ln N$ ) by $2 ln N - \frac{l n l n N + l n 4 π}{2 2 l n N} .$ For $N = 50$ : $2 ln 50 = 2.797$ , and the correction subtracts $\frac{l n ( 3.912 ) + l n ( 4 π )}{2 \cdot 2.797} = \frac{1.364 + 2.531}{5.594} = 0.696$ , giving $2.797 - 0.696 = 2.101$ standardized units. (Here every " $ln$ " is natural log.) Why this step? the crude $2 ln N = 2.80$ overstates the max; the $ln ln N$ term is the standard extreme-value correction and matters a lot at small $N$ .
Inflation $\approx σ \cdot 2.101 = 0.02 \times 2.101 = 0.0420$ , so the reported accuracy is $\approx 0.80 + 0.042 = 0.842$ . Why this step? scale the standardized max back by $σ$ and add the true mean.

Verify: $ln 50 = 3.9120$ ; $2 \cdot 3.9120 = 2.797$ ; correction $= 0.696$ ; corrected $= 2.101$ ; $0.02 \cdot 2.101 = 0.04202$ ; $0.80 + 0.04202 = 0.8420$ ✓. The student reports ~84.2% but truth is 80% — a +4.2-point optimistic bias purely from touching the test set 50 times. (The naive $2 ln N$ estimate would have said +5.6, an overstatement.) Fix: tune on validation, touch test once. This is the parent's headline mistake, quantified.

Recall Self-check

Safe LR range for $L = \frac{1}{2} c θ^{2}$ ? ::: $0 < η < 2/ c$ . What happens exactly at $η c = 2$ ? ::: $r = - 1$ : the weight bounces $\pm$ forever at constant magnitude — never converges, never diverges. Difference between the two "just right" sub-cases? ::: $0 < η c < 1$ shrinks smoothly with no overshoot ( $r \in (0, 1)$ ); $η c = 1$ jumps to the minimum in one step ( $r = 0$ ). New LR when batch size ×4? ::: ×4 (linear scaling rule). Probability a log-uniform sample hits one decade out of 4? ::: $0.25$ . Weight-decay stability condition? ::: $0 < η λ < 2$ (else $∣1 - η λ ∣ \geq 1$ and decay oscillates or blows up). Why not tune on the test set? ::: the max of many noisy test scores is optimistically biased (about +4.2 points here).

Worked examples — Hyperparameter tuning for deep nets