Foundations — Learning rate warmup

This page assumes you have seen nothing. Every letter, arrow, and word used on the parent page Learning rate warmup is unpacked here, in an order where each idea leans only on the ones before it. By the end you should be able to read that page's formulas without pausing on a single symbol.

0. The scene: what is being trained?

Before any symbol, picture the thing we are adjusting.

We collect all the knob values into a single named bundle so we can talk about "the whole model's state" at once.

Why the topic needs it: warmup controls how fast we turn the dials. To talk about turning them we first need one name for "the dials" — that name is $\theta$.

1. Loss $\mathcal{L}$ — the "how wrong am I?" number

The notation $\mathcal{L}(\theta ;\text{batch})$ reads as: "the loss you get when the knobs are set to $\theta$ and you test on this particular scoop of data." The semicolon separates what we tune ($\theta$) from what we happen to be looking at (the batch, defined next).

2. Batch — a scoop of the training data

The picture: a giant pile of flashcards. You don't study all million at once — you draw a stack of, say, 32, quiz yourself, adjust, then draw the next stack. Each stack is a batch; each round is a step.

Why the topic needs it: the parent's Reason 3 (large-batch training, the Linear scaling rule (large batch training)) is about what happens when $B$ gets large — and warmup is what keeps that stable.

3. The step counter $t$ and the warmup length $T_w$

Why the topic needs it: every warmup formula is a function of $t$ compared against $T_w$. These three symbols are the timeline the whole schedule is drawn on.

4. The gradient $g_t$ and the symbol $\nabla$

Here is the tool that tells us which way is downhill.

5. The learning rate $\eta$ — the star of the show

Now we can read the core update rule the parent uses:

${\theta }_{t+1}={\theta }_t-\eta g_t$

Why the topic needs it: warmup is literally a rule for choosing $\eta$ at each step $t$. Everything above was setup so this sentence makes sense.

6. Writing $\eta$ as a function of time: $\eta (t)$

Why the topic needs it: the linear-warmup formula $\eta (t)={\eta }_{\text{peak}}t/T_w$ is nothing but "draw a straight line from ${\eta }_{\text{start}}$ up to ${\eta }_{\text{peak}}$." Recognizing $\eta (t)$ as a curve on a graph is what makes that obvious.

7. Two shapes you'll meet: line and half-cosine

Why the topic needs it: the parent's full recipe is warmup line then cosine slide. You now hold both pieces.

8. One extra name to avoid a classic mix-up: $v_t$

Prerequisite map

Read it top-down: weights and data build the loss landscape; the gradient reads that landscape; the update rule walks it using a stride $\eta$; making $\eta$ depend on step $t$ gives a schedule; warmup is the opening of that schedule, with Adam's $v_t$ and the cosine slide feeding in.

Equipment checklist

Cover the right side and answer before revealing.