Worked examples — Grid search and random search

This page is the "prove it on real numbers" companion to the parent topic. We will not re-explain the theory — instead we hunt down every kind of situation a hyperparameter search can throw at you and grind each one out by hand.

Before we touch a single example, let us agree on the words, because we will use them constantly.

Look at the figure below: it is exactly that map. The bright ridge in the middle is where scores are highest, the white star marks the true best configuration, and each orange dot is one flag a $3\times 3$ grid plants. Notice the grid's dots never land exactly on the star — that gap is the whole story of this page. Keep this map in mind; every later example is a slice or variation of it.

---

The scenario matrix

Every worked example below is tagged with one cell from this table. Together they cover all of it.

Cell	Case class	What is "extreme" about it
A	Low-dimension grid (2 knobs)	The friendly base case — count and cost
B	Sign / boundary of a knob	A hyperparameter at its zero / edge value (e.g. $\lambda =0$, dropout $=0$)
C	High-dimension explosion	Adding one knob — the exponential blow-up
D	Grid vs Random, equal budget	Same number of trials, unique values per axis
E	One knob matters, one is noise	The classic "why random wins" limiting case
F	Log scale vs linear scale	Sampling orders of magnitude, not steps
G	Real-world word problem	Cloud cost & wall-clock time budgeting
H	Exam twist	Cross-validation multiplier + "which method?" trap
I	Where grid wins	Tiny discrete space / very low budget — guaranteed coverage

---

Cell A — the friendly base case

Step 1 — Multiply, don't add. $\text{models}=3\times 4=12$ Why this step? Grid search tries every pairing of a value from list one with a value from list two. For each of the 3 learning rates there are 4 regularizations, giving $3\times 4$. We multiply (not add) because the choices are independent — this is the same reason a menu with 3 mains and 4 drinks gives 12 meals, not 7.

Step 2 — Fold in cross-validation.

$\text{training runs}=12\times 5=60$ Why this step? Each of the 12 configurations must itself be trained 5 times to get one trustworthy score, so we multiply again.

Recall Verify

Total configurations equals product of grid sizes ::: $3\times 4=12$, and $12\times 5=60$ runs. Units: (configs) × (folds) = (runs). ✓

---

Cell B — a knob at its boundary value

Step 1 — Read the role of $\lambda$. In Regularization the training objective is $\text{loss}(\mathbf{w})=\text{data-error}(\mathbf{w})+\lambda \cdot \text{penalty}(\mathbf{w}).$ Why this step? You cannot judge a boundary value without knowing what the symbol multiplies. Here $\lambda$ scales the penalty on large weights.

Step 2 — Set $\lambda =0$. The whole second term vanishes: $\text{loss}(\mathbf{w})=\text{data-error}(\mathbf{w})+0=\text{data-error}(\mathbf{w}).$ Why this step? Substituting the boundary value shows the model with no penalty at all — pure fit-the-data. This is the overfitting extreme, and it is a perfectly valid, informative point on the grid, not a degenerate one. Keep it.

Step 3 — The genuinely degenerate case. Now imagine $\lambda \to \infty$: the penalty dominates, all weights are crushed toward zero, the model predicts a constant (underfitting). That is the useless extreme. Why this step? We push the knob to its other boundary to map out the full behaviour range: at one edge ($\lambda =0$) the model overfits, at the far edge ($\lambda \to \infty$) it underfits. Showing both edges tells the reader the entire span the knob controls, so no limiting value is left unexplored.

Recall Verify

At $\lambda =0$ the objective reduces to ::: just the data-error term (no penalty); it is the overfitting-prone extreme, fully valid to grid over.

---

Cell C — one more knob, exponential pain

Step 1 — Write the general count. For $k$ knobs with $n$ values each: $\text{models}=n^k.$ Why this step? Every knob multiplies the count by its number of values (Cell A logic, repeated), so equal-sized lists give a power.

Step 2 — Plug in. $5^5=3125,5^6=15625.$ Why this step? We substitute the concrete numbers $n=5$, $k=5$ and then $k=6$ into the general formula $n^k$ so the abstract power becomes two countable model totals we can actually compare.

Step 3 — Take the ratio. $\frac{5^6}{5^5}=5.$ Why this step? The ratio isolates the cost of one extra knob. Adding a single hyperparameter multiplies your entire compute bill by 5 (in general, by $n$, the number of values on the new axis). This is the curse of dimensionality in a nutshell.

Recall Verify

Adding one axis with $n$ values multiplies cost by ::: $n$; here $5^6/5^5=5$, and $5^5=3125$, $5^6=15625$.

---

Cell D — equal budget, unequal coverage

Step 1 — Grid's distinct $h_1$ values. A $3\times 3$ grid reuses each $h_1$ value across the 3 settings of $h_2$: $\text{distinct }{h}_1=\sqrt{9}=3.$ Why this step? In a square grid of $n$ points there are $\sqrt{n}$ rows, so only $\sqrt{n}$ unique values per axis.

Step 2 — Random's distinct $h_1$ values. Each of the 9 draws picks $h_1$ freshly from a continuous range, so (with probability 1) all differ: $\text{distinct }{h}_1=9.$ Why this step? We contrast the two counts on the same budget to expose the core trade-off: random spends all 9 evaluations on 9 fresh $h_1$ values, while grid recycles.

Step 3 — The picture. Project all flags onto the east–west axis. The grid's 9 flags land on only 3 columns (heavy stacking); random's 9 land on 9 separate columns.

Why this matters: $9/3=3\times$ more resolution on the axis that might be the important one — for free, same budget.

Recall Verify

For $n$ trials in a square grid, distinct values per axis ::: $\sqrt{n}$ (here 3), while random gives $n$ (here 9).

---

Cell E — the "one knob matters" limit

Step 1 — Evaluate grid's three $h_1$ values. $f(0)=-(0-0.6{)}^2=-0.36,f(0.5)=-(0.5-0.6{)}^2=-0.01,f(1)=-(1-0.6{)}^2=-\mathrm{0.16.}$ Why this step? Since $h_2$ does nothing, only the $h_1$ value decides the score. Grid's best is $h_1=0.5$ with score $-0.01$.

Step 2 — Grid's ceiling. No matter which of the 9 lattice points you pick, $h_1$ is one of $\{0,0.5,1\}$, so the best achievable score is $-0.01$. Grid can never reach $0$ because $0.6$ is not on its lattice. Why this step? We fix the upper bound grid can ever hit on this problem, so we have a concrete number to beat.

Step 3 — Why random can beat it. Random draws 9 different $h_1$ values across $[0,1]$. The chance that at least one lands in the good band, say within $0.05$ of $0.6$ (a band of width $0.1$, i.e. $10\%$ of the range), is $1-(1-0.1{)}^9=1-0.9^9\approx \mathrm{0.6126.}$ Why this step? Each draw independently misses the band with probability $0.9$; all nine miss with probability $0.9^9$; subtract from 1 for "at least one hit." A $\approx 61\%$ chance of landing near the true peak — versus grid's guaranteed miss.

Recall Verify

Grid's best score here is $-0.01$ at $h_1=0.5$, and $P(\text{random hits the }0.1\text{-band in }9\text{ draws})=1-0.9^9\approx 0.613$.

---

Cell F — sampling across orders of magnitude

Step 1 — Uniform on the raw value. The range has total width $0.1-0.0001=0.0999$. The decade $[10^{-4},10^{-3}]$ has width $0.001-0.0001=0.0009$. Fraction: $\frac{0.0009}{0.0999}\approx 0.009=0.9\%.$ Why this step? Uniform sampling spreads points evenly on the number line, so the tiny low decade almost never gets sampled — nearly all points crowd near $0.1$.

Step 2 — LogUniform: uniform on the exponent. Using the recipe above, take ${\log }_{10}$: the range becomes $[-4,-1]$, total width $3$. The decade $[10^{-4},10^{-3}]$ becomes $[-4,-3]$, width $1$. Fraction: $\frac{1}{3}\approx 0.333=33.3\%.$ Why this step? Learning rates matter by their order of magnitude ($0.001$ vs $0.01$ is a real difference; $0.051$ vs $0.052$ is not). Sampling uniformly in $\log$ space gives each decade equal love — see Learning Rate Scheduling for why the scale is multiplicative.

Recall Verify

Under Uniform, the lowest decade gets $\approx 0.9\%$ of samples; under LogUniform it gets $1/3\approx 33.3\%$.

---

Cell G — real-world budget word problem

Step 1 — Count runs. $4\times 3\times 2=24\text{ configs},24\times 5=120\text{ runs}.$ Why this step? This is the base counting from Cell A applied twice: multiply the grid axes to get configurations, then multiply by folds to get actual training runs. Every downstream number rides on this count, so we nail it first.

Step 2 — Total GPU-time. $120\text{ runs}\times 12\text{ min}=1440\text{ min}=24\text{ GPU-hours}.$ Why this step? Cost and time both flow from total compute, so we convert the run count into a single physical quantity — GPU-hours — using $1440/60=24$.

Step 3 — Money. $24\text{ GPU-hours}\times \$2.40/\text{hour}=\$57.60.$ Why this step? Cost tracks total GPU-hours regardless of parallelism (8 GPUs for 3 h costs the same as 1 GPU for 24 h). 57.60 > \40$ — over budget. ✗

Step 4 — Wall-clock with 8 GPUs. $\frac{24\text{ GPU-hours}}{8\text{ GPUs}}=3\text{ hours}.$ Why this step? Runs are independent, so parallelism divides wall-clock time (not cost).

Step 5 — The fix: cap the budget with random search. Replace the exhaustive grid with a random search capped at 60 runs instead of 120: $60\times 12=720\text{ min}=12\text{ GPU-hours},$ $12\text{ GPU-hours}\times \$2.40/\text{hour}=\$28.80,\frac{12}{8}=1.5\text{ hours wall-clock}.$ Why this step? We halve the run count by swapping the exhaustive grid for a capped random budget, because random search can stop at any number of trials while a grid must finish every combination to stay meaningful. This drops cost from \57.60$to$$28.80$(underthe$$40$ limit ✓) and wall-clock from 3 h to 1.5 h — the earlier theory ("random can stop anytime") is exactly what rescues the deadline. See Early Stopping and Bayesian Optimization for even smarter spending.

Recall Verify

Full grid = 120 runs = 24 GPU-hours = $57.60 (over budget), wall-clock 3 h; capped random = 60 runs = 12 GPU-hours = $28.80, 1.5 h.

---

Cell H — the exam twist

Step 1 — Fix part (a). They forgot the fold multiplier: $10\times 10=100\text{ configs},100\times 5=500\text{ trainings}.$ Why this step? "Number of models tried" (100) ≠ "number of times the fit routine ran" (500). Exams love this gap. The reported 100 undercounts compute by $5\times$.

Step 2 — Part (b): match method to space. Checklist from the parent note:

• 6 dimensions (≥4) → favors random.
• Continuous knobs → favors random (no forced discretization).
• One dominant knob suspected → random gives it $n$ distinct values instead of $n^{1/6}$ (Cells D & E).

With $n=64$ trials, grid gives $64^{1/6}=2$ values per axis (useless), while random gives up to 64 distinct values on the important axis. $64^{1/6}=2.$ Why this step? Solving $n^{1/k}$ shows a 6-D grid at 64 trials tests only 2 settings of the crucial knob — catastrophic. So the correct answer is random search (or, for even smarter spending, Bayesian Optimization, with the top configurations combined via ensembling).

Recall Verify

A $10\times 10$ grid with 5-fold CV is $100\times 5=500$ trainings, not 100; and a 6-D grid at 64 trials tests only $64^{1/6}=2$ values per axis.

---

Cell I — the case where grid wins

Step 1 — Count the whole space. $2\times 2=4\text{ combinations}.$ Why this step? When knobs are discrete and few, the space is finite and tiny — the "map" from figure s01 collapses to just 4 lonely points, not a continuous surface.

Step 2 — Grid's guarantee. A grid over all 4 combinations with a budget of 4 evaluates every one exactly once. Probability it misses the best: $P(\text{grid misses best})=0.$ Why this step? Exhaustiveness is grid's superpower — when the whole space fits in the budget, coverage is total and deterministic.

Step 3 — Random's risk on the same budget. Random with replacement might draw duplicates and skip a combination. The chance random's 4 draws cover all 4 distinct combinations is $\frac{4!}{4^4}=\frac{24}{256}=0.09375\approx 9.4\%.$ Why this step? There are $4^4$ equally likely draw-sequences and only $4!$ that hit each combination once; the ratio is the "collect all 4" probability. So random has only a $\approx 9.4\%$ chance of even seeing the best config — a $90.6\%$ chance of missing full coverage. Grid wins decisively here.

Recall Verify

With 4 combinations and 4 trials, grid misses the best with probability $0$; random covers all four with probability $4!/4^4=0.09375$.

---

Recall gauntlet

Prerequisite links for this page: Cross-Validation, Overfitting and Underfitting, Regularization, Learning Rate Scheduling, Training-Validation-Test Split, Early Stopping, Bayesian Optimization, Ensemble Methods, and the parent Grid search and random search.