Instruction tuning: Visual walkthrough — AI-ML deep dive

We assume you know nothing except: a language model reads text left to right and, at every position, guesses what comes next. Everything else — probability, log, cross-entropy, masking — we draw before we use.

Step 1 — What a language model actually outputs

WHAT. At each position in the text, the model does not "know" the next word. It outputs a guess spread over every possible word — a list of numbers, one per word in its vocabulary, all positive and adding up to $1$ . That list is called a probability distribution. A number of $0.7$ next to the word "cat" means "I'm 70% sure the next word is cat."

WHY. We need something we can score. A single guessed word can only be right or wrong (0 or 1), which gives us nothing to nudge. A whole distribution lets us ask "how much probability did you put on the word that actually came next?" — and that number can be pushed up smoothly. Smooth = we can do calculus = we can train.

PICTURE. Look at the bars: the model reads "The mitochondria is the" and outputs a bar chart over candidate next words. The true next word is powerhouse (pale-yellow bar). Training will try to make that yellow bar taller.

Step 2 — A sentence is a chain of these guesses

WHAT. To score a whole sentence, we let the model guess word 1, then word 2 given word 1, then word 3 given words 1–2, and so on. The probability of the entire sentence is all those single-step guesses multiplied together:

$p_{θ} (x_{1}, \dots, x_{T}) = guess for word 1 p_{θ} (x_{1}) \cdot word 2 given word 1 p_{θ} (x_{2} ∣ x_{1}) \dots last word given all before p_{θ} (x_{T} ∣ x_{< T}) = \prod_{t = 1}^{T} p_{θ} (x_{t} ∣ x_{< t}) .$

Term by term: the big $\prod$ ("pi", capital P) just means "multiply all these together", the way $\sum$ means add. Each factor is one bar-height from Step 1. $T$ is how many tokens the sentence has.

WHY multiply? Because "A and B and C all happen" has probability (chance of A) × (chance of B given A) × (chance of C given A,B). This is the chain rule of probability, and it is exact — no approximation. We chose multiplication, not addition, because independent-ish "and"s multiply.

PICTURE. A chain of links: each link is one guess, and the sentence's total score is the product running along the chain.

Step 3 — Products are painful; logarithms rescue us

WHAT. Multiplying hundreds of tiny numbers (each below $1$ ) gives an underflow-tiny result the computer can't handle. So we take the logarithm of the whole product. The magic property of $lo g$ :

$lo g (a \cdot b) = lo g a + lo g b \Rightarrow lo g p_{θ} (x_{1}, \dots, x_{T}) = \sum_{t = 1}^{T} lo g p_{θ} (x_{t} ∣ x_{< t}) .$

The product $\prod$ turned into a sum $\sum$ . Nothing else changed.

WHY the log and not, say, a square root? Because only the logarithm turns "multiply" into "add." That is the exact pain we have (a giant product), so it is the exact tool that cures it. A bonus: $lo g$ of a probability between $0$ and $1$ is a negative number, and it stretches out the near-zero region — so a bar of height $0.001$ (a bad guess) becomes a big-magnitude penalty, while a bar of $0.99$ (a good guess) is nearly $0$ penalty.

PICTURE. The curve $y = lo g x$ on $0 < x \leq 1$ : near $x = 1$ (confident and right) it sits near $0$ ; as $x \to 0$ (confident and wrong) it plunges to $- \infty$ . That plunge is the model's punishment for being sure and wrong.

Step 4 — Flip the sign: from "reward" to "loss"

WHAT. Training machinery is built to minimize things (roll downhill). But we want to maximize the log-probability of the correct words. Same goal, flipped sign — put a minus in front:

$L (θ) = - \sum_{t = 1}^{T} lo g p_{θ} (x_{t} ∣ x_{< t}) .$

This quantity is exactly what is called cross-entropy loss (see Cross-Entropy Loss). $L$ ("script L") is the loss — a number we want as small as possible. Small loss ⇔ the model put high probability on the true words.

WHY minus? Maximizing $lo g p$ and minimizing $- lo g p$ are the same instruction to the optimizer. We flip so the optimizer's built-in "go downhill" points toward "make correct words more probable."

PICTURE. Same log curve as Step 3, mirrored to $- lo g x$ : now it is a valley wall rising to $+ \infty$ near $0$ . Bad guesses = high on the wall = big loss the optimizer wants to slide down from.

Step 5 — The instruction example is two glued pieces

WHAT. Every instruction-tuning example (see Supervised Fine-Tuning (SFT) and Prompt Templates and Chat Formats) is one long token string made of two parts glued together:

$prompt P [### Instruction: Add 2 and 3] response R [### Response: 5] .$

$P$ (prompt) ::: the request. The user types this. The model will never have to invent it.
$R$ (response) ::: the answer. The model must generate this. This is the behavior we want.

WHY split? Because the two parts deserve opposite treatment. There is zero value in teaching the model to predict text the user already provides. All the value is in shaping what it produces after the request.

PICTURE. The glued strip, blue for prompt tokens, pink for response tokens, with the "generation begins here" arrow pointing exactly at the first response token.

Step 6 — The mask: switch off the prompt's grade

WHAT. We attach to each position $t$ a switch $m_{t}$ (the mask):

$m_{t} = {01 if token t is in the prompt P if token t is in the response R .$

Multiply every per-token loss by its switch. Prompt tokens get $\times 0$ (erased); response tokens get $\times 1$ (kept):

$L (θ) = - \sum_{t} m_{t} lo g p_{θ} (x_{t} ∣ x_{< t}) .$

Term by term: $m_{t}$ is the on/off switch, $lo g p_{θ} (\cdot)$ is the per-token reward from Step 3, and the minus makes it a loss (Step 4). Positions where $m_{t} = 0$ contribute nothing, so their gradients vanish — the model is not pushed at all to predict the prompt.

WHY not just delete the prompt? Because the model still needs to read the prompt to answer! The prompt stays in the context $x_{< t}$ (the model looks at it), but it is not graded. Read it, don't get scored on it.

PICTURE. Same glued strip, now with a row of switches underneath — 0s under blue prompt tokens, 1s under pink response tokens — and the graded region shaded.

Step 7 — The degenerate cases (never skipped)

Cover every corner, or the reader hits a wall you didn't draw.

Case A — empty response ( $\sum_{t} m_{t} = 0$ ). Then $\frac{1}{\sum _{t} m _{t}} = \frac{1}{0}$ : undefined. This is a broken training example (nothing to learn from). In practice such rows are dropped. Draw it, don't divide by it.

Case B — empty prompt (all $m_{t} = 1$ ). Every token is graded — this collapses back to ordinary next-token pretraining. Instruction tuning with masking is a strict generalization of pretraining: set the mask all-ones and you recover the base objective.

Case C — perfect model ( $p_{θ} = 1$ on every response token). Then $lo g 1 = 0$ , so $L = 0$ . Loss bottoms out at exactly zero — you cannot do better than certain-and-correct.

Case D — confidently wrong ( $p_{θ} \to 0$ on a true response token). Then $- lo g p_{θ} \to + \infty$ . One arrogantly-wrong answer token can dominate the whole loss. This is why garbage responses in the data teach garbage: the model is dragged hard toward reproducing them.

Step 8 — Why this single mask gives zero-shot obedience

WHAT. Train across diverse tasks (translate, summarize, add) — all wearing the same ### Instruction … ### Response … costume (see FLAN and Zero-shot Generalization and Self-Instruct and Alpaca). Because only the response is graded, the one habit every example reinforces is: "text after ### Response: should satisfy the request, then stop."

WHY it generalizes. The model is never rewarded for a specific task's wording — it's rewarded for the abstract shape "instruction → obey." So a brand-new instruction it has never seen still triggers obey-mode. That is the FLAN headline result.

PICTURE. Three different tasks funnel through the identical template into one shared behavior; a fourth, unseen task ("write a haiku") slides through the same funnel and comes out obeyed.

The one-picture summary

Everything compressed: guesses → chain → log → minus → glue → mask → averaged loss. Follow the arrows; the boxed loss at the end is the same one from Step 6.

Recall Feynman retelling — the whole walkthrough in plain words

A language model, at every spot in a sentence, hands you a betting spread over what word comes next (Step 1). To score a whole sentence, multiply the bets it placed on the words that truly appeared (Step 2). Multiplying hundreds of tiny bets is a numerical nightmare, so we take logs, which turn the multiply into an add and punish confident-wrong bets harshly (Step 3). We flip the sign so "make good bets" becomes "make this number small," which the training machine can roll downhill on — that's cross-entropy loss (Step 4). Now, an instruction example is a question glued to an answer (Step 5). We only want to grade the answer, so we lay a row of on/off switches: off (0) over the question, on (1) over the answer, and multiply each word's score by its switch — the question's grade vanishes, the answer's survives (Step 6). We check the corners: no answer → broken example; no question → plain pretraining; perfect answer → zero loss; confidently-wrong answer → giant loss, which is exactly why bad training answers poison the model (Step 7). Because every task hides behind the same ### Response: costume and only the answer is graded, the model learns one portable habit — obey and stop — so it obeys instructions it has never even seen (Step 8).

Connections

Instruction tuning — the parent topic this walkthrough derives.
Cross-Entropy Loss — Steps 3–4 build exactly this.
Pretraining Objective (Next-token prediction) — the all-ones-mask special case (Step 7B).
Supervised Fine-Tuning (SFT) — the training regime this loss lives in.
Prompt Templates and Chat Formats — the glue of Step 5.
FLAN and Zero-shot Generalization — the payoff of Step 8.
Self-Instruct and Alpaca — where the diverse examples come from.
RLHF and DPO — the stage after this loss has done its job.