Hyperparameter tuning for deep nets: Foundations

This page assumes you have seen nothing. Before you read the parent topic, every letter it throws at you is defined here, in order, each one earning the right to appear before the next uses it.

0. The picture we keep returning to

Everything in this topic is about walking downhill on a landscape. The landscape is the loss, the ground you stand on is the parameters, and the size of each step is the learning rate. Hold that image.

Figure — Hyperparameter tuning for deep nets

Figure 1 (above): the bowl-shaped surface is the loss. A ball rolling to the bottom is training. How big each roll-step is, and how the ball is nudged, is decided by the symbols below. Each figure on this page carries a bold Figure N caption in its title bar so you can always match the text to the picture.

1. $θ$ — the parameters (the "inside dials")

Plain words: the millions of adjustable numbers inside the net.
The picture: a single dot's position on the landscape in Figure 1. Move $θ$ → the dot moves.
Why the topic needs it: the whole point of training is to change $θ$ until the loss is small. Hyperparameters are the rules by which $θ$ is allowed to move.

2. $L$ — the loss (the height of the landscape)

Plain words: the "mistake score."
The picture: the height of the surface in Figure 1 at the dot's location. Low ground = low loss.
Why the topic needs it: training = finding the lowest point of $L$ . Every hyperparameter is judged by how well it helps us get there.

We will meet $L$ in three flavours, distinguished by a subscript:

Symbol	Read as	Computed on
$L_{train}$	training loss	data used to fit $θ$
$L_{val}$	validation loss	held-out data used to choose dials
$L_{test}$	test loss	data touched once, at the end

The reason for three separate landscapes is the whole subject of Cross-Validation and Data Splits and Bias-Variance Tradeoff. Keep them apart; the parent note's biggest warning is about mixing them.

3. $g$ and $\nabla$ — the gradient (which way is downhill)

Before we can step downhill, we need to know which direction is downhill. That is exactly what the gradient tells us.

Plain words: an arrow pointing in the direction of steepest increase of the loss.
The picture: at the ball's feet, an arrow pointing straight uphill; to descend we step the opposite way, $- g$ .
Why this tool and not another? We want the direction that drops the loss fastest. Among all directions, the negative gradient is provably the steepest-descent direction — that is precisely the question "which way lowers $L$ most for a tiny step?" that the gradient answers. Nothing simpler does the job.

Figure 2 (above): the orange arrow ( $g$ , uphill) and the magenta arrow ( $- g$ , downhill). We always walk along magenta.

3b. Two tiny tools before the proof: the dot $\cdot$ and the norm $∥ \cdot ∥$

The next section multiplies two arrows together and measures an arrow's length. Both need defining first.

4. $η$ — the learning rate (how big a step)

Knowing the direction $- g$ is not enough; we must choose how far to move. That distance is the learning rate.

Plain words: step size on the landscape.
The picture: the length of each hop the ball takes in Figure 1.
Why the topic crowns it king: $η$ multiplies every step, so it decides whether the ball settles into the valley or bounces out of it entirely.

Intuition WHY subtracting

$η g$ actually lowers the loss (the Taylor argument) We claimed stepping along $- g$ goes downhill. Here is why, with one tool: the first-order Taylor approximation. For a tiny step $Δ θ$ , the loss changes by approximately its slope times the step (the multiplication is the dot product we just defined): $L (θ + Δ θ) \approx L (θ) + g \cdot Δ θ .$ Why this tool? We can't see the whole landscape, but near the current point the surface looks like a straight ramp of slope $g$ — that is what "first-order" means, and it is exactly enough to predict which direction lowers $L$ . Now plug in our chosen step $Δ θ = - η g$ : $L (θ - η g) \approx L (θ) - η (g \cdot g) = L (θ) - η ∥ g ∥^{2} .$ Since $η > 0$ and $∥ g ∥^{2} \geq 0$ (a squared length is never negative, as defined above), the change $- η ∥ g ∥^{2}$ is never positive — the loss goes down (or stays flat only when $g = 0$ , i.e. we are already at the bottom). That is the guarantee behind the update rule.

Intuition Every case of

$η$ (Figure 3)

$η$ too small (left panel): tiny hops, the ball crawls, may stop on a flat shelf. Loss creeps down forever.
$η$ just right (middle): steps land closer and closer to the bottom — smooth convergence.
$η$ too big (right panel): each hop overshoots the valley and lands higher up the far wall — the loss grows without bound and overflows to Inf (infinity) in the computer's arithmetic. (A true NaN, "Not a Number", is a separate glitch from an invalid operation such as $0/0$ or $- 1$ ; runaway divergence itself produces Inf, which downstream steps can then turn into NaN.)
$η = 0$ : no movement at all, the ball never learns. The Taylor guarantee above only holds for tiny steps; a too-big $η$ leaves the region where the straight-ramp approximation is valid, which is exactly why it can overshoot. More on shrinking $η$ over time lives in Learning Rate Schedules and Warmup.

5. $ar g min$ — "the input that gives the smallest output"

The parent writes $θ^{*} = ar g min_{θ} L_{train} (θ)$ . This looks scary; it is one plain idea.

Plain words: "min" gives you the lowest height; "argmin" gives you the location of the lowest height.
The picture: in Figure 1, $min L$ is the depth of the valley; $ar g min L$ is the spot on the ground directly under it.
Why the topic needs it: it lets us write "training finds the best parameters" in one crisp symbol: $θ^{*} = ar g min_{θ} L_{train}$ .

6. $λ$ — a hyperparameter (an "outside dial")

Before we read the parent's key equation, one piece of notation must be earned: the semicolon inside the loss.

The parent's key equation nests two searches:

\theta^{*}(\lambda)=\arg\min_{\theta}L_{\text{train}}(\theta;\lambda).$$ Read it as two loops: - **Inner loop:** *given* a fixed choice of dials $\lambda$, train to get the best parameters $\theta^*(\lambda)$ — this is ordinary gradient descent on the landscape $\lambda$ shaped. - **Outer loop:** among all dial-settings $\lambda$, pick the one whose trained model scores best on the *validation* landscape. - **Why two nested minimizations?** If we picked $\lambda$ to minimize *training* loss, we'd choose whatever memorizes the data. We need the held-out validation score to reward *generalization*. This split is the heart of [[Bias-Variance Tradeoff]]. --- ## 7. Specific dials you will meet Now that $\lambda$ is defined, here are the concrete dials the parent ranks: | Symbol | Name | What it controls | Deeper note | |---|---|---|---| | $\eta$ | learning rate | step size | [[Learning Rate Schedules and Warmup]] | | $B$ | batch size | how many examples per step | [[Gradient Descent and SGD]] | | $\lambda_{L2}$ | L2 / weight decay | pull of weights toward zero | [[Regularization - L2 and Dropout]] | | $p$ | dropout rate | fraction of neurons switched off | [[Regularization - L2 and Dropout]] | | $D,\,H$ | depth, width | number of layers / neurons | — | | $\beta_1,\beta_2$ | momentum terms | how past gradients are averaged | [[Adam and Momentum Optimizers]] | > [!definition] Batch size $B$ > Instead of computing the gradient $g$ on the *entire* dataset each step (slow) or on one example (noisy), we average $g$ over a **mini-batch** of $B$ examples. Larger $B$ → smoother, less noisy $g$; smaller $B$ → noisier but cheaper steps. See [[Gradient Descent and SGD|Gradient Descent and Stochastic Gradient Descent (SGD)]]. > [!definition] Depth $D$ and width $H$ > A neural net is built from **layers** stacked one after another. $D$ (depth) is the **number of layers** — how many transformation steps the data passes through from input to output. $H$ (width) is the **number of neurons in a layer** — how many parallel units process the data at each step. Both are whole positive numbers (you cannot have $2.5$ layers), typically ranging from a handful to hundreds. > **How they affect capacity:** *capacity* is how complicated a function the net can represent. Increasing $D$ or $H$ adds more weights, so the net can fit more intricate patterns — higher capacity. But higher capacity fits *training* data more easily and so risks **overfitting** (great $L_{\text{train}}$, poor $L_{\text{val}}$), which is why $D$ and $H$ are tuned on validation loss and paired with more regularization when raised. This capacity-vs-generalization tension is the subject of [[Bias-Variance Tradeoff]]. > [!definition] The subscripts $L2$, $\beta_1$, $\beta_2$ > A **subscript** is just a label glued to a symbol to tell two cousins apart. $\lambda_{L2}$ is the weight-decay strength; $\beta_1$ and $\beta_2$ are two different averaging factors used by [[Adam and Momentum Optimizers]]. They are *not* powers — $\beta_1$ is "beta number one," not "beta to the first." --- ## 8. Reading $10^{-3}$ and log scale The parent samples $\eta$ between $10^{-5}$ and $10^{-1}$ on a **log scale**. Two symbols to unpack. > [!definition] Powers of ten $10^{k}$ > $10^{-3}$ means "$1$ divided by $10$ three times" $= 0.001$. The little raised number $k$ is the **exponent**; negative $k$ makes small numbers. Each step in $k$ multiplies by $10$ — a jump of a whole **order of magnitude**. > [!definition] Log scale > A **log scale** places numbers by their *exponent* rather than their raw value, so $0.001,\,0.01,\,0.1$ sit at **equal spacing**. We use it because the effect of $\eta$ is *multiplicative*: doubling $\eta$ matters the same whether $\eta$ is $0.001$ or $0.01$, even though the raw gaps differ tenfold. > [!mnemonic] Reading exponents > The exponent counts zeros (or decimal hops). $10^{-3}=0.001$: three hops of the decimal point to the left. --- ## Prerequisite map ```mermaid graph TD A["Parameters theta W and b"] --> C["Loss L height"] C --> D["Gradient g downhill arrow"] D --> E["Learning rate eta step size"] C --> F["Train val test splits"] E --> G["Update rule theta minus eta g"] F --> H["argmin choose the winner"] G --> I["Hyperparameter lambda outer loop"] H --> I I --> LOG["Log scale search over eta"] LOG --> TUNE["Hyperparameter Tuning"] I --> TUNE ``` Read top-down: parameters give a loss, the loss gives a gradient, the gradient plus a step size gives the update; splits plus argmin let us choose dials; all of it feeds the tuning topic. --- ## Equipment checklist Answer each before moving on. Cover the right side and self-test. What does $\theta$ collect, and where does it live in the landscape picture? ::: All weights $W$ and biases $b$; it is the *position* of the dot on the loss surface. What single number does $L(\theta)$ output, and what does its "height" mean? ::: How wrong the net is; the vertical height of the surface — low = good. What does the gradient $g=\nabla_\theta L$ point toward, and which way do we step? ::: It points *uphill* (steepest increase); we step along $-g$, downhill. What does the dot product $g\cdot v$ compute, and what is $\lVert g\rVert$? ::: $g\cdot v$ multiplies matching entries and sums them (how much two arrows align); $\lVert g\rVert$ is the arrow's length, with $\lVert g\rVert^2=g\cdot g\ge 0$. In $\theta \leftarrow \theta - \eta g$, what role does $\eta$ play and what does `←` mean? ::: $\eta$ is the step size (learning rate); `←` means "becomes / is updated to." Using the Taylor argument, why does stepping along $-\eta g$ lower the loss? ::: $L(\theta-\eta g)\approx L(\theta)-\eta\lVert g\rVert^2$, and $\eta\lVert g\rVert^2\ge 0$, so the loss never rises (falls unless $g=0$). Describe the loss curve for $\eta$ too small, just right, and too big. ::: Too small = creeps down slowly; just right = smooth convergence; too big = overshoots, oscillates or overflows to Inf. What is the difference between $\min_\theta L$ and $\arg\min_\theta L$? ::: $\min$ is the lowest *value*; $\arg\min$ is the *input* $\theta$ that achieves it. What does the semicolon in $L_{\text{train}}(\theta;\lambda)$ separate? ::: Left of `;` = what we minimize over ($\theta$); right of `;` = what is held fixed during that minimization (the hyperparameter $\lambda$). Why are there two nested minimizations in $\lambda^{*}=\arg\min_\lambda L_{\text{val}}(\theta^{*}(\lambda))$? ::: Inner loop trains $\theta$ on training loss for a given $\lambda$; outer loop picks $\lambda$ by validation loss so we reward generalization, not memorization. What do depth $D$ and width $H$ count, and how do they change capacity? ::: $D$ = number of layers, $H$ = neurons per layer; raising either adds weights → more capacity → better training fit but higher overfitting risk. Why sample $\eta$ on a log scale between $10^{-5}$ and $10^{-1}$? ::: $\eta$'s effect is multiplicative, so equal spacing per order of magnitude tests small and large values fairly. Are $\beta_1,\beta_2$ powers of $\beta$? ::: No — the subscripts are labels naming two distinct averaging factors.