Worked examples — Bayesian hyperparameter optimization

This page is the drill hall for Bayesian hyperparameter optimization. The parent note built the machinery: a Gaussian Process surrogate that gives us a mean ${\mu }_n(\lambda )$ and an uncertainty ${\sigma }_n(\lambda )$ at every hyperparameter $\lambda$, plus an acquisition function that turns those two numbers into a single "how worth trying is this point?" score.

Here we do nothing new theoretically — we just hit every case the Expected Improvement (EI) and Upper Confidence Bound (UCB) formulas can throw at you, until no scenario surprises you.

Let us re-anchor the four symbols so nothing is assumed.

We will use the standard-normal reference values (memorise these three, they cover most hand calculations):

$Z$	$\Phi (Z)$	$\varphi (Z)$
$-0.58$	$0.281$	$0.337$
$-0.33$	$0.371$	$0.378$
$0$	$0.500$	$0.399$

(Note $\varphi$ is symmetric: $\varphi (-Z)=\varphi (Z)$, and $\Phi (-Z)=1-\Phi (Z)$.)

---

The scenario matrix

Every worked example below is tagged with the cell it covers. Together they fill the grid.

Cell	What makes it special	Which case of $Z$ / $\sigma$
A. Exploit wins	high mean, low uncertainty, $\mu >f^{+}$	$Z>0$ large
B. Explore wins	low mean but huge uncertainty	$Z<0$, but $\sigma$ big
C. The tie / crossover	two points with nearly equal EI	compare $Z$ vs $\sigma$
D. Zero-uncertainty (degenerate)	${\sigma }_n=0$ at an already-sampled point	$\sigma =0\Rightarrow \text{EI}=0$
E. Predicted-worse but positive EI	$\mu <f^{+}$ yet EI $>0$	$Z<0$, $\sigma >0$
F. UCB vs EI disagree	same two points, different acquisition	$\kappa$ tradeoff
G. Limiting behaviour	$\sigma \to 0$ and $\sigma \to \infty$	asymptotics of EI
H. The knife-edge $\mu =f^{+}$	predicted mean exactly equals the best	$Z=0$, $\sigma >0$
I. Real-world word problem	log-scale learning rate, cost budget	end-to-end loop
J. Exam twist	minimisation problem disguised as maximisation	sign flip of $f$

---

Worked examples

Example 1 — Cell A: exploitation dominates

Forecast: the mean already beats the best by a comfortable $0.05$ and there's barely any doubt. Guess: EI is roughly the raw improvement $0.05$, dominated by the first term.

1. Compute $Z$. $Z=\frac{0.90-0.85}{0.02}=2.5$. Why this step? $Z$ tells us the predicted improvement is $2.5$ uncertainty-steps above the current best — deep into "almost certainly better" territory.
2. Look up normal values. $\Phi (2.5)=0.9938$, $\varphi (2.5)=0.0175$. Why this step? $\Phi (2.5)\approx 1$ means ~99% of plausible outcomes beat $f^{+}$; the tiny $\varphi$ says almost no probability sits at the threshold.
3. Plug into EI. $\text{EI}=(0.90-0.85)(0.9938)+(0.02)(0.0175)=0.0497+0.00035=\mathrm{0.0500.}$ Why this step? Exploitation term $0.0497$ swamps exploration term $0.00035$ — exactly Cell A.

Verify: EI ($0.0500$) is essentially the raw improvement $0.05$, as forecast. Sanity: since $Z$ is large positive, $\text{EI}\to \mu -f^{+}$, which is what we got.

---

Example 2 — Cell B: exploration dominates

Forecast: the mean is below the best — pure exploitation would say "skip it". But $\sigma$ is enormous. Guess: EI is still sizeable, maybe even competitive.

1. Compute $Z$. $Z=\frac{0.78-0.85}{0.20}=-0.35$. Why this step? Negative $Z$ = we predict worse than best on average. But only $0.35$ uncertainty-steps below — the bell curve still pokes well above $f^{+}$.
2. Normal values. $\Phi (-0.35)=0.3632$, $\varphi (-0.35)=0.3752$. Why this step? $\Phi (-0.35)=0.36$ says ~36% chance of beating $f^{+}$ despite the low mean; $\varphi$ is near its peak, so exploration term will be large.
3. Plug in. $\text{EI}=(0.78-0.85)(0.3632)+(0.20)(0.3752)=-0.0254+0.0750=\mathrm{0.0496.}$ Why this step? The first (exploitation) term is negative — it penalises the low mean — but the exploration term $0.075$ from high $\sigma$ rescues it.

Verify: $\text{EI}({\lambda }_b)=0.0496$, essentially tied with $\text{EI}({\lambda }_a)=0.0500$. So a confident +0.05 improvement and a wild guess with big uncertainty are worth almost the same — that is Bayesian optimization refusing to be greedy. Sanity: EI $\ge 0$ always (improvement can't be negative), and indeed $0.0496>0$. ✓

---

Example 3 — Cell C & the crossover figure

Forecast: since ${\lambda }_b$ is already almost tied at $\sigma =0.20$, a slightly bigger uncertainty should tip it over.

1. Restate the target line. From Example 1, ${\lambda }_a$ has the fixed value ${\text{EI}}_a=(0.90-0.85)\Phi (2.5)+0.02\varphi (2.5)=0.0500$. We treat this as a constant to beat. Why this step? So the reader need not flip back — the crossover is "when does the varying EI reach $0.0500$?"
2. Set EI as a function of ${\sigma }_b$. With ${\mu }_b-f^{+}=-0.07$ fixed, $Z=-0.07/{\sigma }_b$, so ${\text{EI}}_b({\sigma }_b)=-0.07\Phi \left(\frac{-0.07}{{\sigma }_b}\right)+{\sigma }_b\varphi \left(\frac{-0.07}{{\sigma }_b}\right).$ Why this step? We freeze the mean and let only uncertainty grow — isolating the exploration effect.
3. Read the crossover. Numerically, ${\text{EI}}_b=0.0500$ near ${\sigma }_b\approx 0.202$; the figure below shows ${\text{EI}}_b$ rising through the flat ${\text{EI}}_a$ line. Why this step? Locating the intersection gives the exact uncertainty at which the "explore" pull overtakes a confident $+0.05$ improvement — the whole point of Cell C is to quantify that tipping point, not just assert it exists.

The red curve is the explore point's EI as its uncertainty ${\sigma }_b$ grows; the dashed black line is the fixed exploit EI of $0.0500$. Where red rises above black, exploration wins.

Verify: at ${\sigma }_b=0.20$ we computed ${\text{EI}}_b=0.0496<0.0500$; at ${\sigma }_b=0.25$, $Z=-0.28$, $\Phi =0.3897$, $\varphi =0.3836$, giving ${\text{EI}}_b=-0.07(0.3897)+0.25(0.3836)=-0.0273+0.0959=0.0686>0.0500$. So the crossover lies between $0.20$ and $0.25$ — consistent with the red curve piercing the black line in the figure. ✓

---

Example 4 — Cell D: the degenerate zero-uncertainty point

Forecast: we already know the answer there — there's nothing to learn. Guess: EI $=0$.

1. Find ${\sigma }_n(0.05)$. For a noiseless GP, the posterior variance at an observed input is $0$: ${\sigma }_n(0.05)=0$. Why this step? The GP interpolates its own data exactly, so it has zero doubt where it has measured.
2. Apply the definition. The EI formula's second branch is $\text{EI}=0$ whenever ${\sigma }_n=0$. Why this step? If there's no uncertainty, there is no random upside to expect — improvement is fully determined ($\mu =y_3=0.80<f^{+}$, so improvement is exactly $0$).

Verify: $\max (0,0.80-0.85)=0$, and with $\sigma =0$ there's no distribution to average over, so $\text{EI}=0$. This is why Bayesian optimization never re-samples an exact known point — a crucial safety property. ✓

---

Example 5 — Cell E: predicted-worse yet still worth trying (the parent's Example 1 finished properly)

Forecast: both means are below $0.85$, so both $Z<0$ — pure Cell E. The one with more uncertainty should score higher.

1. Point $\lambda =0.005$: $\mu =0.83,\sigma =0.06$. $Z=\frac{0.83-0.85}{0.06}=-0.333$. $\Phi (-0.333)=0.3696$, $\varphi (-0.333)=0.3778$. $\text{EI}=(-0.02)(0.3696)+(0.06)(0.3778)=-0.00739+0.02267=\mathrm{0.01528.}$ Why this step? Small uncertainty ⇒ small exploration reward; net EI is modest.
2. Point $\lambda =0.08$: $\mu =0.78,\sigma =0.12$. $Z=\frac{0.78-0.85}{0.12}=-0.583$. $\Phi (-0.583)=0.2800$, $\varphi (-0.583)=0.3369$. $\text{EI}=(-0.07)(0.2800)+(0.12)(0.3369)=-0.0196+0.04043=\mathrm{0.02083.}$ Why this step? Double the uncertainty ⇒ the exploration term $0.0404$ dominates, lifting EI above point 1.

Verify: $0.0153$ vs $0.0208$ match the parent's $0.0154$ and $0.02$ (tiny rounding). The higher-EI point $0.08$ is chosen — exactly the parent's conclusion that "exploration wins" even though $0.78<0.83<0.85$. ✓

---

Example 6 — Cell F: UCB and EI can disagree

Recall $\text{UCB}(\lambda )={\mu }_n(\lambda )+\kappa {\sigma }_n(\lambda )$ — a linear mix, no bell-curve mathematics.

Forecast: UCB weights $\sigma$ by a fixed $\kappa =2$; the $0.08$ point has more $\sigma$, so it may win here too — but let's check whether the mean gap flips it.

1. UCB at $0.005$: $0.83+2(0.06)=0.83+0.12=0.95$. Why this step? UCB asks "optimistically, how high could this be?" — mean plus two std-devs.
2. UCB at $0.08$: $0.78+2(0.12)=0.78+0.24=1.02$. Why this step? Same optimism rule; the bigger $\sigma$ inflates the ceiling more.
3. Compare. $1.02>0.95$ ⇒ UCB also picks $0.08$. Agreement with EI here. Why this step? The acquisition rule is "pick the highest score", so comparing the two UCB values is the decision — and with $\kappa =2$ the extra uncertainty of $0.08$ outweighs its lower mean, matching EI's choice.

Now the twist — set $\kappa =0.5$ (a cautious explorer):

• UCB$(0.005)=0.83+0.5(0.06)=0.86$; UCB$(0.08)=0.78+0.5(0.12)=0.84$. Now $0.005$ wins — UCB flips!

Verify: EI is fixed and always preferred $0.08$; UCB's choice depends on $\kappa$. This is the key lesson of Cell F: the acquisition function is a design choice, and UCB's tradeoff is tunable and explicit where EI's is baked in. ✓

---

Example 7 — Cell G: limiting behaviour of EI

Forecast: with a positive mean gap, tiny uncertainty should give EI $\to 0.03$ (we're sure it's better by 0.03); huge uncertainty should make EI grow without bound.

1. As $\sigma \to 0^{+}$: $Z=0.03/\sigma \to +\infty$, so $\Phi (Z)\to 1$, $\varphi (Z)\to 0$. $\text{EI}\to (0.03)(1)+\sigma \cdot 0=\mathrm{0.03.}$ Why this step? Perfect certainty of a $+0.03$ improvement gives EI exactly $0.03$ — no randomness left.
2. As $\sigma \to \infty$: $Z\to 0$, $\Phi (0)=0.5$, $\varphi (0)=0.399$. $\text{EI}\to (0.03)(0.5)+\sigma (0.399)\to \infty .$ Why this step? Infinite uncertainty ⇒ infinite upside potential; EI grows linearly in $\sigma$ (slope $\varphi (0)=0.399$).

The red curve is EI against uncertainty $\sigma$ for a fixed predicted advantage of $+0.03$; the dotted line marks the left-hand floor $0.03$, and the dashed line is the right-hand linear growth with slope $0.399$.

Verify: At $\sigma =0.01$: $Z=3$, $\Phi =0.99865$, $\varphi =0.00443$; EI $=0.03(0.99865)+0.01(0.00443)=0.02996+0.0000443\approx 0.0300$ ✓. At $\sigma =10$: $Z=0.003$, EI $\approx 0.03(0.5012)+10(0.399)=0.015+3.99=4.005$ — enormous, confirming the divergence. The red curve in the figure flattens to $0.03$ on the left and rises linearly on the right. ✓

---

Example 8 — Cell H: the knife-edge $\mu =f^{+}$

Forecast: the exploitation term vanishes (mean gap is $0$), so only the exploration term survives. Guess: EI is small but strictly positive, driven entirely by uncertainty.

1. Compute $Z$. $Z=\frac{0.85-0.85}{0.10}=0$. Why this step? At the knife-edge the predicted improvement is exactly zero uncertainty-steps — the bell curve is centred right on $f^{+}$, so exactly half its mass lies above.
2. Normal values at $Z=0$. $\Phi (0)=0.500$, $\varphi (0)=0.399$ (the peak of the bell curve). Why this step? $\Phi (0)=0.5$ says "50% chance of beating the best"; $\varphi (0)$ is the maximum possible height, so the exploration term is as large as it can be for this $\sigma$.
3. Plug into EI. $\text{EI}=(0.85-0.85)(0.500)+(0.10)(0.399)=0+0.0399=\mathrm{0.0399.}$ Why this step? The first (exploitation) term is exactly $0$ — the mean gives no help at the knife-edge — leaving pure exploration $\sigma \varphi (0)=0.10\times 0.399$. This gives the clean closed form $\text{EI}=0.399\sigma$ whenever $\mu =f^{+}$.

Verify: EI $=0.0399>0$ — even a point predicted to just tie the best is worth trying, because half its uncertainty band lies above $f^{+}$. Sanity check the closed form: $\sigma \varphi (0)=0.10\times 0.399=0.0399$ ✓, matching the full plug-in. ✓

---

Example 9 — Cell I: the real-world word problem (full loop)

Forecast: we did exactly these EI numbers in Example 5 — the edge point $x_b$ wins.

1. Work in $x={\log }_{10}\lambda$, not $\lambda$. Best so far $f^{+}=0.85$. Why this step? Learning rates span five orders of magnitude; on a linear axis the RBF kernel's Euclidean distance $\Vert x-x^{\prime }\Vert$ would treat $0.001$ and $0.01$ as "almost equal", crushing the low end. Taking ${\log }_{10}$ spaces them evenly so the kernel's smoothness assumption is meaningful — see 3.4.2-Gaussian-Processes.
2. EI at $x_a=-2.3$: identical to Example 5 point $0.005$ ⇒ $\text{EI}=0.0153$. Why this step? Acquisition only ever needs $(\mu ,\sigma ,f^{+})$; the physical units of $x$ drop out.
3. EI at $x_b=-1.1$: identical to Example 5 point $0.08$ ⇒ $\text{EI}=0.0208$. Why this step? Same formula, higher uncertainty — exploration lifts this candidate.
4. Decide. Higher EI ⇒ train at $x_b=-1.1$, i.e. $\lambda =10^{-1.1}\approx 0.0794$. Why this step? We spend our single expensive run where the model expects to learn the most.

Verify: $10^{-1.1}=0.07943$ ✓, matching the parent's ${\lambda }_4\approx 0.08$ that then scored $y_4=0.87$ (new best). Contrast with 2.6.11-Grid-Search/2.6.12-Random-Search, which would have spent that run blindly. ✓

---

Example 10 — Cell J: the exam twist (minimisation in disguise)

Forecast: we must flip the sign so "improvement" means "lower loss". Guess: define $g=-\text{loss}$ and everything else follows.

1. Negate the objective. Let $g(\lambda )=-\text{loss}(\lambda )$; maximising $g$ = minimising loss. Why this step? Our EI formula rewards increasing $\mu$; feeding it $-\text{loss}$ makes it reward decreasing loss.
2. Best in $g$-space. Best observed loss $=\min \{0.30,0.45,0.28\}=0.28$, so $f^{+}=-0.28$. Candidate mean in $g$-space $=-0.31$, $\sigma =0.05$ (unchanged — variance is sign-blind). Why this step? $\sigma$ measures spread; negating a variable doesn't change its spread.
3. Compute $Z$ and EI. $Z=\frac{-0.31-(-0.28)}{0.05}=\frac{-0.03}{0.05}=-0.60$. $\Phi (-0.60)=0.2743$, $\varphi (-0.60)=0.3332$. $\text{EI}=(-0.03)(0.2743)+(0.05)(0.3332)=-0.00823+0.01666=\mathrm{0.00843.}$ Why this step? Standard EI in the negated space; the negative first term reflects the predicted loss being worse than the best.

Verify: EI $=0.00843>0$: even though predicted loss $0.31$ is above the best $0.28$, the uncertainty leaves room to beat it — a legitimate exploration candidate. If you had forgotten the sign flip you'd have maximised loss and chased the worst configs. Sanity: $Z<0$ because the mean loss exceeds the best, exactly as expected. ✓

---

Recall

Every cell of the matrix, in one line each. When does EI reduce to the raw mean gap? ::: When $\sigma \to 0^{+}$ with $\mu >f^{+}$: $\text{EI}\to \mu -f^{+}$ (Cell G / Ex 1). Why is EI exactly $0$ at an already-sampled noiseless point? ::: ${\sigma }_n=0$ there, and the formula's second branch sets $\text{EI}=0$ (Cell D / Ex 4). Can a point with mean below $f^{+}$ still be chosen? ::: Yes — if its $\sigma$ is large enough the exploration term outweighs the negative exploitation term (Cell B/E). What is EI at the knife-edge $\mu =f^{+}$? ::: Exactly $\sigma \varphi (0)=0.399\sigma$ — pure exploration, strictly positive (Cell H / Ex 8). How do EI and UCB differ in their explore/exploit control? ::: EI bakes the tradeoff into $\Phi ,\varphi$; UCB exposes it as a tunable $\kappa$ (Cell F). How do you use maximisation-EI for a minimisation objective? ::: Optimise $g=-\text{loss}$; then $f^{+}=-\min (\text{loss})$ and $\sigma$ is unchanged (Cell J).

---

Where to go next: the same surrogate-plus-acquisition loop, applied to whole network structures, is 2.6.15-Neural-Architecture-Search.