Teacher forcing: Visual walkthrough — AI-ML deep dive

Step 1 — What is a sequence model actually doing?

WHAT. A sequence model is a machine that reads a list of things one at a time (letters, words, notes) and, after each one, guesses the next thing. Picture reading the sentence "I am ___" — a good guesser says "happy" or "here", not "purple".

WHY start here. Every equation on this page is really just this one act — guess-the-next-thing — repeated. If you see the guessing machine clearly, everything else is bookkeeping.

PICTURE. In the figure, a box (the model) eats one token, remembers it in a small memory pocket (its hidden state), and points an arrow at its next guess. The pocket is what lets step 2 know what happened in step 1.

Step 2 — Chain the guesses: the sequence probability

WHAT. The chance the model gives to a whole sequence is the product of the chance it gives to each next token, one after another:

$p (y ∣ x) = multiply over every position t = 1 \prod T this token p (y_{t} ∣ all earlier tokens y_{1 : t - 1}, the input we condition on x)$

Reading it term by term: $\prod$ means "multiply everything after it together"; $t = 1$ to $T$ means "for every position from the first to the last ( $T$ = how many tokens)"; $y_{1 : t - 1}$ is shorthand for "the whole list $y_{1}, y_{2}, \dots, y_{t - 1}$ " — the history.

WHY multiply (and why this exact split)? This is the chain rule of probability: the chance of a full story = chance of word 1 $\times$ chance of word 2 given word 1 $\times \dots$ . We use it because our machine can only ever answer one question — "what's next given the past?" — so we must break the whole sequence into exactly those one-step questions.

PICTURE. A ribbon of boxes: each box's guess-arrow feeds forward, and underneath, the little probabilities multiply into one number for the whole sentence.

Step 3 — Turn the product into a sum with a log (and mind the sign)

WHAT. Multiplying many small numbers (each below 1) gives a tiny, fragile number. So we take the logarithm, which turns products into sums. Define the log-likelihood:

$L = lo g p (y ∣ x) = \sum_{t = 1}^{T} model’s own log-probability lo g p_{θ} (y_{t} ∣ y_{1 : t - 1}, x)$

Term by term: $L$ (script L) is the log-likelihood we try to make as large as possible; $\sum$ is "add up" (the log turned the $\prod$ into it); $p_{θ}$ means "the probability as computed by the model whose knobs are $θ$ " — $θ$ (theta) is the bag of all the model's adjustable numbers (weights).

WHY the log — and why this tool, not another? Two reasons, both concrete: (1) $lo g (a \cdot b) = lo g a + lo g b$ , so a fragile product becomes a stable, addable sum a computer won't round to zero; (2) $- lo g p$ is the cross-entropy loss we already minimise (just shown), so we get our training target for free. No other function turns products into sums and matches the loss — that's why it's the log and nothing else.

PICTURE. Left: tiny probabilities shrinking toward zero as we multiply. Right: their logs — negative bars — that we simply stack up.

Reveal — one question:

What does taking the log convert a product into?

A sum (of log-probabilities); its negative is the cross-entropy loss

J

we minimise.

Step 4 — The fork in the road: what do we feed as history?

WHAT. Every step needs the history $y_{1 : t - 1}$ . But we have two possible histories to hand the model:

Teacher forcing: feed the true earlier tokens $y_{1 : t - 1}$ (from the training data).
Free running: feed the model's own earlier tokens $\overset{y}{^}_{1 : t - 1}$ .

The hat on $\overset{y}{^}$ means "produced by the model, not given" — it is the model's own output, which may be wrong.

Definition How the model turns its output into a token

The model outputs a probability over the whole vocabulary. To get an actual token $\overset{y}{^}_{t - 1}$ from it, there are two common choices:

Greedy / $ar g max$ : $\overset{y}{^}_{t - 1} = ar g max (output_{t - 1})$ — pick the single highest-probability word. Deterministic. Used in our worked trace for simplicity.
Sampling: draw $\overset{y}{^}_{t - 1} \sim output_{t - 1}$ — roll a weighted die over the vocabulary, so lower-probability words sometimes appear. This is what real generation often does (it adds variety).

Either way the picked token is a discrete choice, so the map "output $\to$ token" is not differentiable — a gradient cannot flow back through the pick. That is precisely why free-running training is hard (Step 6) and why fixes either avoid it (teacher forcing) or use special estimators.

WHY this is THE decision. During training we own the correct answer, so we get to choose. During inference (real use) we have no choice — there is no answer key, so we must free-run (greedily or by sampling). This mismatch is the whole drama of teacher forcing.

PICTURE. A single arrow splitting into two coloured paths at position $t$ : the magenta path pulls the true previous token down into the box; the orange path loops the model's own previous token back in.

Step 5 — Teacher forcing: every step gets a clean slate

WHAT. With teacher forcing we always take the magenta path. Trace "HELLO" from the parent note (each per-step number below is a loss term $- lo g p$ ):

$L_{T F} = \sum_{t = 1}^{T} (- lo g p_{θ} (y_{t} ∣ always the TRUE letters y_{1 : t - 1}, x))$

Even if the model was only 70% sure of H at $t = 1$ , at $t = 2$ we still hand it the real H. Its mistake is graded but never fed forward.

WHY it trains fast and cleanly. Because the input at every step is fixed truth, the steps are independent to feed forward — no error can leak from one step into the next, and the gradient does not have to crawl back through a chain of non-differentiable picks. Every step starts from a perfect, correct context.

PICTURE. Five boxes in a row. Each box's input arrow (magenta) comes straight from the true-letter track above; the model's guesses (thin orange, below) are only scored, never reused.

Step 6 — Free running: one slip cascades

WHAT. Now take the orange path everywhere. The moment a pick (whether $ar g max$ or a sample) lands on the wrong letter, that wrong letter becomes the next input, and the model is suddenly in a situation it never practised:

$L_{inf} = \sum_{t = 1}^{T} (- lo g p_{θ} (y_{t} ∣ model’s OWN, possibly wrong, letters \overset{y}{^}_{1 : t - 1}, x))$

WHY it goes wrong. Following the parent's greedy trace: at $t = 4$ the model picks O instead of L (argmax $= 0.4$ ). At $t = 5$ it now conditions on its own bad O — a state it was never trained on — and the loss for the true O rockets to $- lo g (0.2) = 1.61$ . Error feeds error. (Sampling can make this worse on any given roll — a low-probability wrong word may be drawn even when the argmax is right.)

PICTURE. The same five boxes, but the input arrows now loop back from each box's own pick. At $t = 4$ a red wrong-guess arrow diverts the track, and every box after it drifts off the true rail — a visible cascade.

Step 7 — The gap, and the compromise dial $ϵ$

WHAT. Teacher forcing optimises the loss under clean histories but at inference we live under own-histories. The difference between these two is exposure bias. The cure is a dial $ϵ \in [0, 1]$ (epsilon), the teacher forcing ratio: at each step, flip a biased coin — with chance $ϵ$ take truth, else take the model's own token.

$input_{t} = {y_{t - 1} \overset{y}{^}_{t - 1} with probability ϵ (truth) with probability 1 - ϵ (own token)$

WHY a dial and not a switch? Extremes are bad: $ϵ = 1$ trains fast but never practises recovery; $ϵ = 0$ practises recovery but barely learns because early training is chaos. A dial that starts near 1 and decays lets the model first learn the pattern, then rehearse fixing its own slips — this is scheduled sampling, a cousin of curriculum learning.

PICTURE. A slider from $ϵ = 0$ (all orange / own tokens) to $ϵ = 1$ (all magenta / truth), with the training loss low on the right, the train-test gap wide on the right and shrinking as we ease left.

The one-picture summary

Everything above is one river with a fork. The chain rule breaks a sentence into next-token guesses (Steps 1–2); the log makes them addable and its negative is the loss (Step 3); the fork chooses which history feeds each guess (Step 4); truth-feeding trains fast but hides mistakes (Step 5); own-feeding is honest but cascades (Step 6); the dial $ϵ$ blends them, at the cost of biased gradients (Step 7).

Recall Feynman retelling — say it like you'd tell a friend

Imagine teaching a kid to recite "HELLO". Teacher forcing is when, after every letter, you say the correct next letter out loud so the kid always continues from the right place — fast learning, but the kid never learns what to do after its own stumble. Free running is letting the kid recite alone — either always saying its most-confident letter (greedy) or sometimes rolling a die over letters (sampling); one wrong letter and the whole word derails, because it's now continuing from a mess it made. The fix is a dial: early on, help almost every time; later, sometimes let the kid continue from its own letter so it learns to recover — but that helping-die makes the learning signal a little crooked, so it's a trade. The math is only bookkeeping for this: multiply the chance of each next letter (chain rule), take logs so the numbers stay sane, and remember the loss is the negative of that log-likelihood — a sum of $- lo g p$ , which for a one-hot target is exactly the cross-entropy. Then choose, at each step, whether the true letter or the kid's own letter is the input. Teacher forcing = truth every time; the gap between "always helped in practice" and "alone in the exam" is exposure bias.

Recall Key clozes

The training technique of feeding the true previous token at every timestep is teacher forcing. The chain rule lets us write $p (y ∣ x) = \prod_{t} p (y_{t} ∣ y_{1 : t - 1}, x)$ so a sequence becomes next-token guesses. The input $x$ is the source sequence we condition on, always given up front. The likelihood is $L = lo g p$ ; the loss we minimise is its negative ( $J = - L$ ), a sum of $- lo g p$ . For a one-hot target, $- lo g p$ equals the cross-entropy of the true and model distributions. Feeding the model's own token is called free running (by argmax or by sampling), and a single mistake cascades. The gap between train-time (truth) and test-time (own tokens) histories is exposure bias. The teacher forcing ratio == $ϵ$ dials between the two; decaying it (scheduled sampling) trades exposure bias for biased gradients==.

See also: 3.6.2-Seq2seq-architecture · 4.2.3-Autoregressive-models · 3.5.13 Teacher forcing (Hinglish)