Foundations — Bayesian hyperparameter optimization

Before you can read the parent note Bayesian hyperparameter optimization, you need to earn every symbol it throws at you. This page defines them one at a time, from absolute zero, in the order you must learn them. Nothing here assumes you've seen the notation before — line one starts from nothing.

0. The thing we are trying to do

A machine-learning model has knobs you set before training — how fast it learns, how much it distrusts complexity, and so on. These knobs are hyperparameters. Turning the knobs changes how well the trained model does on unseen data. We want the knob-setting that does best.

1. The symbol $\lambda$ — one knob-setting

The picture: think of $\lambda$ as a pin dropped on a map. One knob → the pin lives on a line. Two knobs → the pin lives on a flat sheet.

Why the topic needs it: the whole search happens on this map. Everything else is about deciding where on $\Lambda$ to drop the next pin.

2. The symbol $f(\lambda )$ — the hidden score-landscape

Because every setting has a score, we can imagine a landscape: over each point of the map $\Lambda$ there is a hill-height $f(\lambda )$. The best setting is the top of the tallest hill.

Two more symbols travel with $f$:

3. Noise: the symbol $ϵ$ and why $y\ne f$

When we train twice with the same $\lambda$ we don't get exactly the same score — random shuffling, initialization, etc. wobble it. So the measured score $y$ is the true height plus a wobble:

$y=f(\lambda )+ϵ$

Picture: $f(\lambda )$ is the true ground level; each measurement $y$ is a slightly jittered flag stuck near the ground. We only ever see the flags, never the ground itself.

4. Bell curves: $\mathcal{N}$, $\mu$, $\sigma$, ${\sigma }^2$

To reason about "how sure are we?" we need the shape of uncertainty: the bell curve (normal distribution).

Why the topic needs it: at every unseen setting the method reports its belief about $f(\lambda )$ as a bell curve — a best guess $\mu$ and an honesty band $\sigma$. That band is what makes exploration possible.

5. The two ready-made bell-curve tools: $\varphi$ and $\Phi$

The Expected-Improvement formula in the parent note uses two functions of the standard bell curve (the one centred at $0$ with spread $1$).

Why the topic needs them: $\Phi (Z)$ answers "what's the chance we beat the current best?" and $\varphi (Z)$ measures "how big is the surprise if we do?" — the two ingredients of Expected Improvement.

6. Vectors and matrices: bold lists and grids

With many past trials we stack them.

Picture: a vector is a single shelf of boxes; a matrix is a whole cabinet of boxes with rows and columns. The GP posterior formulas multiply these to combine everything we've measured.

7. Subscripts and the data set ${\mathcal{D}}_n$

8. The surrogate and the acquisition function (names only, here)

Two big ideas the parent builds fully — here we just name the boxes so the symbols aren't mysterious:

The explore–exploit balance in one line:

• Exploit = trust high ${\mu }_n$ (go where we already think it's good).
• Explore = chase high ${\sigma }_n$ (go where we're unsure — it might be secretly great).

This same tension is the heart of multi-armed bandits.

How the foundations feed the topic

Compare the naive alternatives that ignore the surrogate: Grid Search and Random Search. The grown-up version that searches over whole model designs is Neural Architecture Search.

Equipment checklist

Test yourself — say the answer out loud before revealing.