Grid search and random search: Visual walkthrough

Everything below assumes you know only what a graph and a fraction are. We define every other symbol on the spot.

Step 1 — What a "hyperparameter" even is, as a knob

WHAT. A model has knobs you must set before training starts. The parent calls these hyperparameters. Call the first knob $h_{1}$ and the second knob $h_{2}$ . Think of $h_{1}$ as "learning rate" and $h_{2}$ as "batch size" — but for now they are just two dials.

WHY. To search a space you must first see the space. Two knobs means a 2D plane: the horizontal axis is every value $h_{1}$ can take, the vertical axis every value $h_{2}$ can take. One dot in that plane = one full model you would train — and, using the convention above, that dot is $h = (h_{1}, h_{2})$ .

PICTURE. Below, the flat plane is the search space $H$ (a fancy name for "all the dot-positions you're allowed to pick"). We have picked no dots yet — it is empty on purpose.

Step 2 — What "score" is, drawn as height above the plane

WHAT. Train a model at a point $h = (h_{1}, h_{2})$ , check it on held-out data (see Cross-Validation), and you get a number: the validation score. We call this number $s (h)$ — read "the score at point $h$ ". Higher $s$ = better model.

WHY. If every point has a height $s (h)$ , the whole plane becomes a landscape — hills and valleys. Our goal, in the parent's notation, is

h^{*} = ar g h \in H max s (h) .

Read this term by term:

$s (h)$ — the height (score) at knob-setting $h = (h_{1}, h_{2})$ .
$max$ — "the biggest height".
$ar g max$ — not the biggest height itself, but the location $h$ where that biggest height sits. ( $ar g$ = "argument", i.e. the input.)
$h^{*}$ — that winning location. The star means "best".

PICTURE. The landscape below has one tall ridge. Notice the ridge runs along the $h_{2}$ direction — meaning $s$ barely changes as $h_{2}$ moves. That is the crucial fact we exploit in Step 4.

Step 3 — Grid search: lay the budget on a lattice

WHAT. Grid search chops each axis into a few fixed values and trains one model at every crossing. With 3 values for $h_{1}$ and 3 for $h_{2}$ we get $3 \times 3 = 9$ points — a tidy lattice.

WHY. It is the obvious, exhaustive plan: try every combination, guarantee you find the best point on the lattice. For $k$ knobs with $n_{i}$ values each the count is

Total models = i = 1 \prod k n_{i},

where $\prod$ means "multiply them all together" and $n_{i}$ is how many values you allowed on knob $i$ .

PICTURE. Nine blue dots, perfectly aligned. Look at the shadows dropped onto the bottom axis: all three dots in a column land on the same $h_{1}$ value. So the grid only ever tests 3 distinct $h_{1}$ values, even though it spends 9 runs.

Step 4 — The trap: important vs unimportant knobs

WHAT. Recall the ridge from Step 2 ran along $h_{2}$ . That means $h_{1}$ is the important knob (moving it changes the score a lot) and $h_{2}$ is unimportant (moving it barely matters). To model this cleanly, we write the same score surface $s (h)$ from Step 2 in a suggestive split form — call it $f$ , but keep in mind $f \equiv s$ (they are the identical surface, just relabelled to expose its structure):

= s (h) f (h_{1}, h_{2}) = g (h_{1}) + ϵ (h_{2}),

term by term:

$f (h_{1}, h_{2})$ — exactly the score $s (h)$ of Step 2, nothing new; the letter $f$ just flags that we are now studying its shape.
$g (h_{1})$ — the real signal; a sharp peak somewhere along $h_{1}$ .
$ϵ (h_{2})$ — near-flat "noise" from the unimportant knob.

WHY. Finding $h^{*}$ really means finding the peak of $g$ — a 1D problem hiding inside our 2D picture. To resolve a sharp 1D peak you need many distinct $h_{1}$ values. But Step 3 showed the grid gives only 3. If the peak sits between two grid columns, the grid never sees it.

PICTURE. The score curve $g (h_{1})$ is drawn along the bottom. The three grid columns (blue) sample it at only three spots and miss the true peak (dashed red). All three models inside one column are near-duplicates — wasted runs.

Step 5 — Random search: scatter the same budget

WHAT. Random search draws each of the 9 points from a distribution over the whole rectangle. No two points are forced to share an $h_{1}$ column, so their shadows spread out along the important axis.

WHY. Each of the 9 runs spends its $h_{1}$ coordinate on its own random draw. For a continuous distribution the drawn values are distinct with probability 1 (formally "almost surely" — two continuous draws landing on the exact same real number has probability zero; in practice they are always different). So in expectation you test $n$ distinct $h_{1}$ values with $n$ runs — versus only $n^{1/ k}$ for the grid. For the important knob, random search buys you $9$ probes instead of $3$ .

PICTURE. Nine orange dots scattered. Drop their shadows onto the $h_{1}$ axis: nine distinct marks, densely covering the important direction. The chance one of them lands near the true peak is far higher.

Step 6 — Edge case: when the grid actually wins

WHAT. Random search is not magic. Two degenerate situations flip the verdict.

WHY & PICTURE (case A — both knobs matter equally). If there is no unimportant direction — the peak is a tight bump in 2D, not a ridge — then even coverage matters and a grid's uniform spacing can be competitive, while random points may clump and leave gaps. (Left panel below: random points leave a hole right where the bump sits.)

WHY & PICTURE (case B — genuinely discrete knobs). If a knob is a choice, like optimiser $\in {adam, sgd}$ , there is nothing "between" the options to sample. A grid lists each label exactly once if you set its value-list to the labels themselves. Note the qualifier: if your budget is larger than the number of labels, a naive grid or random sampler will re-visit labels — a plain random draw over ${adam, sgd}$ just re-picks one of the two — unless you post-process to keep only unique label settings. The honest takeaway: for a pure discrete axis, enumerate the labels once; don't let either method waste runs re-testing them. (Right panel: two vertical lines, no continuum.)

Step 7 — Edge case: mixed continuous–discrete spaces (the real world)

WHAT. Most real tuning jobs are neither all-continuous nor all-discrete: e.g. learning rate $α$ is continuous, but optimiser $\in {adam, sgd}$ and "use dropout?" $\in {yes, no}$ are discrete labels.

WHY. The right strategy is a hybrid, and the picture tells you how to split your budget:

On each continuous axis, sample randomly — you reap the "distinct values" win from Step 5.
On each discrete axis, enumerate the labels (grid-style) — one setting each, no waste (Step 6, case B).

So for a space with $d$ discrete labels total and continuous axes underneath them, you effectively run random search inside each of the $d$ discrete buckets. Your budget $n$ splits as roughly $n / d$ random continuous probes per bucket. This is exactly what libraries do when they accept a distribution for some knobs and a fixed list for others.

PICTURE. Two vertical bands (the two optimiser labels). Inside each band the continuous knob $α$ is sampled randomly — dense coverage where it matters, one clean choice where it doesn't.

Step 8 — Degenerate budget: what happens at $n = 1$ and $n \to \infty$

WHAT. Push the budget to its limits.

WHY.

$n = 1$ : a grid must pick one corner or centre — a fixed guess. Random picks one random point — same odds on a flat surface, but on a ridge random at least varies the important knob. Neither is reliable; you learn almost nothing. Pair either with Early Stopping to avoid wasting even that one run on a hopeless region.
$n \to \infty$ : both methods densely cover $H$ and both find $h^{*}$ . The whole debate is a finite-budget phenomenon. When runs are free, method choice is irrelevant.

PICTURE. As $n$ grows, the "distinct $h_{1}$ values" curves for grid ( $n^{1/ k}$ , blue) and random ( $n$ , orange) both rise — but random's straight line towers over the grid's flattening root. The visible gap between the curves is the wasted budget.

The one-picture summary

The single frame above stacks the whole argument: same 9-run budget, same ridge landscape. Grid (blue) tests 3 $h_{1}$ values and misses the peak; random (orange) tests 9 and catches it. The side histograms show coverage of the important knob — the real currency of tuning.

Recall Feynman retelling — say it back in plain words

We had two dials to tune and a score that goes up and down as we turn them. We can't feel the slope, so we just have to try dial-settings and look at the score. We only get 9 tries. If we line the tries up in a neat 3×3 grid, then — because three tries share each column — we only ever test three settings of the left dial. But it turned out the left dial is the one that matters (a ridge, not a bump). Three probes of the important dial is too coarse; we sail right past the best spot. If instead we throw the 9 tries down at random, every try uses a new value of the left dial — nine probes of the thing that matters — so we're far likelier to hit the peak. The catch: if both dials mattered equally, or a dial is just a menu of names, the neat grid is fine or even better. Real problems mix the two — so sample the continuous dials randomly and just list the menu dials once each. And if tries were free, it wouldn't matter at all — this is entirely a story about spending a small budget wisely.

Grid tests how many distinct values of one knob, given $n$ runs over $k$ knobs? ::: $n^{1/ k}$ Random tests how many distinct values of that knob (in expectation)? ::: $n$ Why are the random draws "distinct"? ::: For a continuous distribution they are distinct with probability 1 (almost surely) — exact ties have probability zero Which landscape shape makes random search win most? ::: A ridge — one knob important, others nearly flat How do you handle a mixed continuous–discrete space? ::: Random-sample the continuous knobs, enumerate the discrete labels once each (hybrid) Why can't we just use gradient descent on the score? ::: The validation score has no usable gradient — each point costs a full training and the surface is noisy

Related building blocks: Overfitting and Underfitting and Regularization are why these knobs exist; Training-Validation-Test Split and Cross-Validation are how each score is measured; Learning Rate Scheduling, Ensemble Methods and Bayesian Optimization are where you go next.