Grid search and random search: Question bank — AI-ML deep dive

Before we start, two words we lean on constantly:

Hyperparameter = a knob you set before training (learning rate, regularization $λ$ , tree depth). Contrast with a parameter = a number learned during training (a weight).
Budget = the total number of models you can afford to train. Every trap below is really about "where do I spend my budget?"

True or false — justify

TF — "Grid search is guaranteed to find the globally best hyperparameters."

False. It only guarantees the best point on the grid you defined; the true optimum can sit between grid lines and be missed entirely.

TF — "Random search can beat grid search even with the exact same budget."

True. With

n

trials, random search tries

n

distinct values along each axis, while a grid tries only

n^{1/ k}

per axis — so on the important axis random search samples far more finely.

TF — "Because random search uses randomness, its result is not reproducible."

False. Fix the random seed and every sampled configuration is identical run-to-run; reproducibility comes from the seed, not from being deterministic-by-design.

TF — "More hyperparameters in a grid always means better tuning."

False. Each new axis multiplies the model count; most of that extra compute lands in obviously-bad regions, so cost explodes faster than quality improves.

TF — "Grid search adapts: once it sees a bad region it stops searching there."

False. Grid search is blind and non-adaptive — it evaluates every listed point regardless of what earlier points revealed.

TF — "If two hyperparameters do not interact, grid search wastes almost nothing."

False. Even with no interaction the grid re-tests the unimportant axis many times, spending trials that reveal no new information about the important axis.

TF — "Random search naturally handles continuous hyperparameters; grid search must discretize them."

True. Random search samples from a distribution over a continuous range, whereas a grid can only list a finite set of chosen values.

TF — "A 5-fold cross-validation multiplies the number of models trained, not the number of configurations."

True. You still evaluate the same set of configurations, but each one is trained 5 times (once per fold), so training runs = configurations

\times

folds.

TF — "Random search always outperforms grid search."

False. For very few, discrete hyperparameters (say optimizer = adam/sgd), a small grid is exhaustive and cheap, and random sampling could even miss a valid option.

Spot the error

Error — "I have 6 hyperparameters, so I'll grid 5 values each — only a few hundred models."

Wrong count:

5^{6} = 15, 625

models, not hundreds. Cost is multiplicative across axes, so six axes is already infeasible.

Error — "Random search with 60 trials tries only 60 unique learning-rate values total because some repeat."

In expectation all 60 sampled values are distinct on a continuous range (probability of an exact repeat is essentially zero), so it explores ~60 unique values per axis, not fewer.

Error — "I'll grid learning rate over {0.001, 0.01, 0.1} on a linear scale."

Those points are already log-spaced; the mistake is instead sampling a linear-uniform range like Uniform(0.001, 0.1), which crams almost all samples near the top and starves the small-rate region. Use a log-uniform range for scale-spanning knobs.

Error — "I found the best grid point, so I'll report its validation score as my final performance."

The validation score is optimistically biased because you selected on it. Report performance on a held-out test set the search never touched. (See Training-Validation-Test Split.)

Error — "I tuned on the whole dataset, then measured accuracy on that same dataset."

That leaks the answer — you must tune with Cross-Validation on train/validation splits and keep a final test set unseen, or you cannot detect overfitting.

Error — "Grid search wastes resolution, so I'll just add more values everywhere."

Adding everywhere multiplies cost again. The real fix is unequal resolution — many values on the important axis, few on the rest — which random search does automatically.

Error — "I'll pick the config with the lowest training loss from the search."

Lowest training loss usually means most overfit. Select on validation score, which estimates how the model generalizes.

Why questions

Why — "Why can't we just use gradient descent to optimize hyperparameters like we do for weights?"

The validation loss is not differentiable with respect to hyperparameters (each evaluation requires a full retrain), so there is no gradient to follow — hence we search instead of descend.

Why — "Why does random search gain so much specifically in high dimensions?"

A grid's values-per-axis,

n^{1/ k}

, shrinks fast as the number of axes

k

grows, so the grid becomes coarse; random search keeps

n

distinct values per axis regardless of

k

Why — "Why is 'some hyperparameters matter more than others' the key assumption behind random search's advantage?"

If the objective is roughly

f (h_{1}, h_{2}) = g (h_{1}) + noise (h_{2})

, only fine sampling of

h_{1}

helps; random search spends its distinct samples on

h_{1}

while a grid wastes them re-testing

h_{2}

Why — "Why should you retrain on train+validation combined after choosing the best config?"

More data usually gives a slightly better final model; you used the split only to choose the config, and now you want the strongest model from that choice before the test evaluation.

Why — "Why are both grid and random search easy to parallelize?"

Every configuration is trained independently with no shared state, so trials can run simultaneously on separate cores/machines.

Why — "Why can random search 'stop anytime' but a fixed grid cannot cleanly?"

Random trials are exchangeable — the first 40 of 60 are already a valid random sample — whereas a grid only becomes a complete sweep once every listed point is done.

Edge cases

Edge — "What happens with a grid that has a single value for every hyperparameter?"

It trains exactly one model — you've fixed the config, not searched, so no tuning occurs.

Edge — "What if the true optimum lies exactly between two grid points?"

Grid search can never reach it; you'd need a finer grid, a shifted grid, or random/Bayesian sampling to land near it.

Edge — "What if two hyperparameters strongly interact (best

C

depends on

γ

)?"

You must test combinations, not each axis alone — evaluating

C

at a fixed bad

γ

tells you nothing, which is exactly why both methods sweep joint points.

Edge — "What if all hyperparameters are equally important with no noise dimension?"

Random search's advantage shrinks — a well-placed grid covers a low-dimensional, uniformly-important space about as well.

Edge — "What if your search budget is 1 trial?"

Grid degenerates to one arbitrary corner; random search gives one random draw. Neither tunes meaningfully — you need enough trials to see the performance surface vary.

Edge — "What if the validation set is tiny?"

Scores become noisy, so the 'best' config may just be lucky; use Cross-Validation to average over folds and reduce that selection noise.

Edge — "Random search sampled a learning rate so large the model diverged (loss = NaN)."

Treat it as a failed trial with worst-possible score; this is normal — a wide distribution should occasionally probe unstable regions, and pairing search with Early Stopping limits the wasted compute.

Recall One-line summary to self-test

Grid = exhaustive, deterministic, explodes exponentially, coarse per-axis. Random = sampled, seed-reproducible, scales to high dimensions, fine per-axis. Both need honest validation via a held-out split. Which method gives more distinct values per axis for the same budget? ::: Random search — $n$ distinct values versus a grid's $n^{1/ k}$ .