Bayesian hyperparameter optimization: Visual walkthrough

Before any symbols: three plain-word characters run through this whole page.

The knob — one number we can dial. We call it $λ$ (lambda), just a name for "the setting".
The score — how good the model is at that setting (say validation accuracy). We call it $f (λ)$ .
The doubt — how unsure we are about the score at settings we have not tried yet.

That's it. No probability theory assumed. We build the rest.

Step 1 — The thing we cannot see

WHAT. There is a true curve $f (λ)$ : for every knob-setting on the horizontal axis, it tells you the score on the vertical axis. We do not get to see this curve. We only get to poke it at a few places, and each poke is expensive.

WHY. This is the whole reason Bayesian optimization exists. If pokes were free we would poke everywhere (2.6.11-Grid-Search) or poke at random (2.6.12-Random-Search). They are not free, so every poke must be chosen with care.

PICTURE. The dashed grey curve is the hidden truth. The three yellow dots are the only three places we have poked. Notice: between and beyond the dots, we are guessing.

Figure — Bayesian hyperparameter optimization

Step 2 — Turning dots into a belief (mean + doubt band)

WHAT. We replace "three lonely dots" with a belief: a smooth guessed curve $μ_{n} (λ)$ (our best guess of the score everywhere) plus a shaded band $σ_{n} (λ)$ (how wide our doubt is everywhere).

WHY this and not just connect-the-dots? A plain interpolation gives one line and pretends it is certain. But far from any dot we are not certain, and we must remember that — because doubt is exactly what makes a place worth exploring. The tool that carries both a guess and a doubt at once is the Gaussian Process (full machinery in 3.4.2-Gaussian-Processes). For this page you only need its two outputs.

PICTURE. The blue line is $μ_{n} (λ)$ . The blue shaded region is the doubt band $μ_{n} (λ) \pm σ_{n} (λ)$ . Look at the two behaviours you must never forget:

At a yellow dot the band pinches to zero width — we poked here, so no doubt.
Far from every dot the band balloons — we are guessing, so lots of doubt.

Step 3 — Marking the champion so far

WHAT. Among the scores we have actually measured, one is the highest. Call it $f^{+}$ (read "f-plus"), the incumbent best.
$f^{+} = max (y_{1}, y_{2}, \dots, y_{n})$

Here

max (\dots)

pick the largest of the measured scores.

And

f^{+}

that largest value — the bar every future candidate must beat.

WHY. "Improvement" only has meaning against a reference. $f^{+}$ is that reference: the height to clear. Anything below it is not an improvement at all.

PICTURE. The red horizontal line sits at $f^{+}$ , level with the tallest yellow dot. Everything above the red line is "better than what we already have."

Step 4 — "Improvement" is a one-sided ruler

WHAT. Suppose the true score at a candidate $λ$ turned out to be some value. The improvement is how far above the red line it lands — and if it lands below the line, the improvement is just zero (a worse setting doesn't hurt us; we simply don't keep it).
$I (λ) = max (0, f (λ) - f^{+})$

f (λ) - f^{+}

how far above the champion this candidate would be.

max (0, \cdot)

floor it at zero — negative improvement counts as none.

WHY the floor at zero? Because we keep the best we ever saw. A flop costs us the training time but does not lower our final answer. So the downside is capped while the upside is open. This asymmetry is the engine of the whole method — it means bold, uncertain guesses are cheap in expectation.

PICTURE. The green wedge is $I$ : it rises above the red line and is flattened to zero below it. That hard corner at the red line is the $max (0, \cdot)$ doing its job.

Step 5 — But we don't know $f (λ)$ , only a bell of possibilities

WHAT. At an untested $λ$ we do not have a single value of $f (λ)$ . The Gaussian Process says $f (λ)$ could be any value, more likely near $μ_{n} (λ)$ , less likely far from it — a bell curve of width $σ_{n} (λ)$ .

WHY a bell curve? Because that is what the Gaussian Process promises: at each setting the possible scores are normally distributed. So instead of one improvement, we have a whole spread of possible improvements. We must average them.

PICTURE. A vertical bell sits on top of the candidate $λ$ . The part of the bell poking above the red line is exactly the improvement region. The part below the line collapses to zero improvement.

Step 6 — Averaging the bell: Expected Improvement is born

WHAT. We average the improvement over the whole bell. That average is the Expected Improvement, $EI (λ)$ — one number per candidate telling us how good a bet it is.

WHY average, and why does it split into two pieces? The improvement region has two natural contributions: (1) how high the centre of the bell already sits above the line, weighted by how often we clear the line, and (2) bonus mass leaking over the line from the bell's spread even when the centre is below it. These are exactly the two terms below.

First a shorthand — how many "doubt-widths" the centre sits above the champion:
$Z = \frac{μ _{n} ( λ ) - f ^{+}}{σ _{n} ( λ )}$

μ_{n} (λ) - f^{+}

how far our guess beats the champion (can be negative).

\div σ_{n} (λ)

measured in units of doubt-width, so it is comparable everywhere.

Now the result. We need two helper curves of the standard bell:

Where the formula comes from (in one honest line). We integrate the green wedge of Step 4 against the bell of Step 5. Substituting $z = \frac{u - ( μ _{n} - f ^{+} )}{σ _{n}}$ turns the integral into $\int z ϕ + \int ϕ$ , and the standard fact $\int z ϕ (z) d z = - ϕ (z)$ produces exactly the two boxed terms.

PICTURE. Two candidate settings side by side. Candidate A has a high guess but a thin band — its EI is nearly all left term (exploit). Candidate B has a lower guess but a fat band — its EI is nearly all right term (explore). EI lets us compare them on one scale.

Step 7 — The degenerate case: $σ_{n} = 0$

WHAT. Exactly at a place we already poked, the doubt vanishes: $σ_{n} (λ) = 0$ . The formula's top line would divide by zero, so we use the bottom line: $EI = 0$ .

WHY zero and not something? If there is no doubt, there is no bell — the score is pinned to the single value we already measured. A known value cannot surprise us upward, so re-poking it can never improve anything. EI correctly refuses to waste a poke there.

PICTURE. At the yellow dot the band pinches shut, the bell collapses to a spike, and the EI curve drops to the floor. Every acquisition function must handle this pinch — it is what stops the optimizer from testing the same point forever.

Step 8 — Pick the peak, poke it, repeat

WHAT. Compute $EI (λ)$ across the whole knob range. The setting with the highest EI is our next poke:
$λ_{n + 1} = ar g max_{λ} EI (λ)$

ar g max

the location of the maximum, not the maximum value itself.

WHY. EI already folded exploration and exploitation into one score. Trusting its peak means we automatically chase promising-and-good or promising-and-uncertain regions, whichever wins. After poking, the band pinches there, EI drops there, and next round the peak moves elsewhere — the search naturally spreads out. (This exploit/explore tension is the same one studied in 4.3.1-Multi-Armed-Bandits, and the loop is the backbone of 2.6.15-Neural-Architecture-Search.)

PICTURE. Bottom panel: the full EI curve with its tallest peak circled. A red arrow drops from that peak up to the setting we will test next. That is one full turn of the loop.

Step 9 — A number you can check by hand

WHAT. Take the parent note's worked case: candidate $λ = 0.005$ with $μ_{n} = 0.83$ , $σ_{n} = 0.06$ , champion $f^{+} = 0.85$ .

$Z = \frac{0.83 - 0.85}{0.06} = \frac{- 0.02}{0.06} \approx - 0.333$

Standard-bell lookups: $Φ (- 0.333) \approx 0.3696$ and $ϕ (- 0.333) \approx 0.3773$ .

$EI = (0.83 - 0.85) (0.3696) + (0.06) (0.3773)$ $= - 0.00739 + 0.02264 \approx 0.0152$

Notice the left term is negative (our guess is below the champion) yet the right term rescues it — the doubt spills enough mass over the bar to make the candidate mildly worth trying. That is exploration paying rent, exactly as the two-term split predicted.

PICTURE. The final composite: mean line, doubt band, red champion line, the specific bell at $λ = 0.005$ , and the two coloured contributions ( $- 0.0074$ and $+ 0.0226$ ) stacked to show the net $+ 0.0152$ .

The one-picture summary

Top: the belief (blue mean + doubt band), the yellow pokes, and the red champion line $f^{+}$ . Bottom: the EI curve derived from them, with its peak marking the next poke. The whole method is just this pair of panels, redrawn every round.

Recall Feynman retelling — say it like you'd explain to a friend

We're turning a knob and each test costs a fortune, so we can't test everywhere. After a few tests we draw a smooth guess-line through our results, and — crucially — we draw a fuzzy band around it that's thin where we tested and fat where we didn't. We mark the best score we've seen so far with a red line. Now for any untested knob-setting, imagine a little bell of "what the score might be." We only care about the slice of that bell poking above the red line, because a bad result never hurts us — we just don't keep it. We average that above-the-line slice; that average is the Expected Improvement. It's naturally the sum of two urges: "this looks high, go grab it" plus "this is uncertain, go find out." We compute EI for every setting, jump to the tallest EI, pay for one test, and repeat. Each test flattens the band and the EI right where we looked, so the search keeps wandering to fresh, promising places until it homes in on the best knob.