Foundations — Instruction tuning

Before you can read the parent note, you need to be fluent in a small pile of symbols and pictures. This page builds every single one of them from zero, in the order they depend on each other. Nothing here assumes you have seen probability, logarithms, or subscripts before.

1. A token — the atom of everything

Think of a sentence being chopped into LEGO bricks. "Add 2 and 3" might become the bricks Add, 2, and, 3. Each brick is a token.

Why do we need this idea? Because every later symbol ($x_t$, the product, the loss) is defined per token. If you don't picture the sentence as a row of numbered bricks, none of the subscripts will mean anything.

2. The subscript $x_t$ — "the token at position $t$"

$t$ is short for "time step" — it moves left to right like a clock ticking once per brick. So $x_1$ is the first brick, $x_2$ the second, and $x_T$ the last (we use a capital $T$ for the total number of tokens in the sequence — the final tick).

3. The shorthand $x_{<t}$ — "everything before slot $t$, in order"

Order matters here — "dog bites man" and "man bites dog" are the same bricks in a different order and mean opposite things, so $x_{<t}$ is a sequence, never a shuffled pile. Picture a cursor blinking between two bricks. Everything to the left of the cursor, read in order, is $x_{<t}$; the very next brick you are about to guess is $x_t$.

Why the topic needs it: the whole job of a language model is "guess $x_t$ given $x_{<t}$." That phrase — the next brick given all previous bricks in order — is the seed of every formula on the parent page.

4. Probability $p$, and conditional probability $p(x_t\mid x_{<t})$

Picture the model as a weather forecaster. It doesn't say "the next word IS the." It says "given everything so far, the has a $0.4$ chance, a has $0.2$, cheese has $0.001$…" — a whole spread of guesses summing to $1$.

The parameters $\theta$ (Greek "theta") are just all the tunable knobs inside the model — the billions of numbers that decide those probabilities. Writing $p_{\theta }$ reminds us "this forecast depends on the current knob settings." Training = turning the $\theta$ knobs.

5. The big sum $\sum$ and the big product $\prod$

We need both: $\prod$ builds the probability of a whole sentence (multiply the guesses), and $\sum$ builds the loss (add up the surprises). Introducing them together now means no summation symbol will surprise you later.

The parent writes the sentence probability with $\prod$: $p_{\theta }(x_1,\dots ,x_T)={\prod }_{t=1}^T{p}_{\theta }(x_t\mid x_{<t}).$

Read it in plain words: "the chance of the whole sentence = (chance of brick 1) × (chance of brick 2 given brick 1) × (chance of brick 3 given bricks 1–2) × …". You build the sentence one guess at a time and multiply the guesses.

6. The logarithm $\log$ — turning products into sums

So, using the magic property term-by-term: $\log p_{\theta }(x_1,\dots ,x_T)={\sum }_{t=1}^T\log p_{\theta }(x_t\mid x_{<t}).$ The scary product became a tidy sum.

7. Negative log-likelihood — the loss (and cross-entropy)

We said $\log p$ is always $\le 0$. If the model is confident and correct, $p\approx 1$ and $\log p\approx 0$ — tiny loss. If it's confident and wrong, $p\approx 0$ and $\log p\to -\infty$ — huge loss.

We want a number that is big when wrong, small when right, so we flip the sign. First the raw quantity, the total negative log-likelihood (NLL) — literally the summed surprise over every token: $\underset{\text{a sum: add up the surprise at each token}}{\underbrace{{\mathcal{L}}_{\text{total}}(\theta )=-\sum _{t=1}^T\log p_{\theta }(x_t\mid x_{<t})}}.$

This total grows just because the sentence is long, which makes long and short examples hard to compare. So we usually report the average NLL — divide by the number of tokens counted: ${\mathcal{L}}_{\text{avg}}(\theta )=-\frac{1}{T}{\sum }_{t=1}^T\log p_{\theta }(x_t\mid x_{<t}).$

8. The mask $m_t$ — grade only the answer

Now the one idea that makes instruction tuning different from plain pretraining.

Picture a highlighter running over only the answer half of each example. Highlighted = graded. Un-highlighted = ignored.

The parent's boxed formula is now fully readable — it is the average NLL from Section 7, but averaged over only the highlighted tokens: $\mathcal{L}(\theta )=-\frac{1}{{\sum }_t{m}_t}{\sum }_t{m}_t\log p_{\theta }(x_t\mid x_{<t}).$

• $m_t\cdot \log p_{\theta }(\dots )$ — multiplying by $0$ deletes prompt tokens from the sum; multiplying by $1$ keeps response tokens.
• ${\sum }_t{m}_t$ — just counts how many tokens got highlighted (how many response tokens). Dividing by it gives the average surprise per answer-token, so a long prompt can't drown out a short answer.

How these feed the topic

Everything on the left is a prerequisite for the parent topic, which lives at node I. The base guessing machine (nodes A–F) is exactly the Pretraining Objective (Next-token prediction); the mask (node H) is what turns that pretraining loss into instruction tuning.

Equipment checklist

Cover the right side and test yourself. You are ready when you can answer every line instantly.