Foundations — Grid search and random search

This page assumes you have seen none of the notation in the parent note. We build every symbol, one at a time. By section 6 the parent's headline sentence will read like plain English — but we will not write that sentence down until every piece inside it has been earned.

1. A model is a machine with two kinds of knobs

Picture a coffee machine. Some settings the machine figures out on its own while brewing (how hot the water gets as it flows). Other settings you turn before you press start (bean coarseness, cup size). A machine-learning model is exactly like this: it has knobs it learns for itself, and knobs you must set first.

Figure s01 — A box labelled "the model." On the left, a teal knob marked "parameter" with an arrow reading "learned during training" (the machine turns it). On the right, a burnt-orange knob marked "hyperparameter alpha" with an arrow reading "set by hand first" (you turn it). The picture makes the split concrete: two knobs, two people turning them.

Why does the topic need this split? Because grid search and random search only touch the hand-turned knobs. The learned knobs take care of themselves; the whole tuning problem is about picking good values for the hand-turned ones. See Overfitting and Underfitting for what goes wrong when those hand-turned knobs are set badly.

2. The symbols for the knobs

Now we can name things. A single letter is just shorthand for "the value on one knob." The parent note uses one symbol for learned knobs and several for hand-turned ones — here is every one, spelled out.

You do not need to master SVMs or trees to follow this page — just remember that each of $\alpha ,\lambda ,C,\gamma ,d,n$ is one more hand-turned dial, a single number you choose before training.

3. Bundling knobs into one object: $\mathbf{h}$

A real model has several hand-turned knobs at once. Writing $(\alpha ,\lambda ,C,\dots )$ every time is clumsy, so we bundle them into one ordered list.

Picture $\mathbf{h}$ as a single point in a room. If there are two knobs ($k=2$), the room is a flat floor and $\mathbf{h}$ is a dot on it: left–right = knob 1, forward–back = knob 2. Every possible setting of the two knobs is a different dot on that floor.

Figure s02 — A flat floor with the horizontal axis labelled "knob 1: alpha" and the vertical axis "knob 2: lambda." Several plum dots are scattered on it; one is highlighted in orange and labelled "this dot = configuration h = (alpha, lambda)." The caption at top reads "the whole floor of dots = script-H." This shows that one configuration is one point, and the collection of all points is the space we meet next.

4. The space of all settings: $\mathcal{H}$

If every configuration $\mathbf{h}$ is a dot, then the whole floor of possible dots needs a name.

Why does the topic need this? Grid search and random search are two different ways of picking dots out of $\mathcal{H}$. Grid search lines the dots up neatly; random search scatters them. You cannot describe either method until you have named the room they explore.

5. Measuring how good a setting is: the score

Trying a setting is pointless unless you can grade it. But grade it on which data?

Why validation and not training? Grading on the same data the model learned from would reward memorisation — see Overfitting and Underfitting. To make the grade trustworthy we often average several validation piles, which is Cross-Validation.

6. Picking the winner: $\arg \max$

We have a room of dots ($\mathcal{H}$) and a way to grade each dot (${\text{score}}_{\text{val}}$). The last symbol simply says "keep the best dot."

Now — and only now, with every piece defined — the parent's headline equation reads as plain English:

Left to right: "The best configuration ${\mathbf{h}}^{\ast }$ (the star means optimal) is the one setting $\mathbf{h}$, chosen from all allowed settings $\mathcal{H}$, whose validation score is highest." That is the entire goal of the topic in one line.

Figure s03 — A coloured contour map over the same alpha–lambda floor: brighter regions = higher validation score. A teal star sits on the brightest peak, labelled "best setting h = argmax of score." The picture shows that $\arg \max$ is simply "point at the tallest spot," and grid/random search are two ways of hunting for that peak without a map.*

7. Counting the work: the product symbol $\prod$

Grid search tries every combination, so we must count them.

8. Sampling instead of listing: distributions

Random search does not list values; it draws them at random from a described range.

Why this tool and not plain uniform for $\alpha$? A learning rate of $0.05$ and $0.06$ behave almost identically, but $0.05$ and $0.005$ behave completely differently. Log-uniform spreads the darts where the behaviour changes, not where the numbers change. (Categorical knobs from section 4 are "sampled" too — you just pick one name from the menu uniformly.)

How the foundations feed the topic

The map below is read top-to-bottom: each box is a foundation from this page, and the arrows show which foundation you need before the next makes sense. The two left-hand foundations about knobs feed the idea of a configuration $\mathbf{h}$, which feeds the space $\mathcal{H}$. The space then branches three ways: counting its size (the product $\prod$) leads to grid search; sampling from it (distributions) leads to random search; and grading points in it (validation score → $\arg \max$) is what both methods ultimately optimise. If any single box is still fuzzy, that is exactly the prerequisite to revisit before opening the parent note.

Equipment checklist

Cover the right side and test yourself before reading the parent note.