Foundations — Teacher forcing

Before you can understand that sentence fully, you need to know what a "token", a "sequence", a "probability of the next thing", a "product over time", and a "log-loss" actually are — as pictures, not just symbols. This page builds every one of them from zero, in the order the parent note secretly assumes them.

1. A token, and a sequence of tokens

Read the notation slowly:

• $y$ — the whole list (the finished sentence "HELLO").
• $y_t$ — the token sitting at position $t$ (the little subscript $t$ is just an address, like a house number on a street).
• $T$ — the total length, the house number of the last token.

The picture: a row of boxes, left to right, each box holding one token. Position matters — "am I" and "I am" use the same tokens but are different sequences.

Why the topic needs it. Teacher forcing is entirely about what we put in each box's slot at training time. If you don't picture the row of boxes with numbered positions, none of the later equations mean anything.

2. "Given the past" — conditional probability

The model does not guess $y_t$ out of thin air. It guesses it given everything before it.

Two new pieces of notation, defined:

• $y_{1:t-1}$ — shorthand for the slice $(y_1,y_2,\dots ,y_{t-1})$, i.e. all tokens before position $t$. The colon : means "from ... up to".
• $p(\cdot )$ — a probability: a number between $0$ and $1$ saying "how likely". $0$ = never, $1$ = certain, $0.7$ = fairly likely.

Why "given" and not just $p(y_t)$? Because language is context-dependent. After "HELL" the next letter is almost surely "O"; on its own, "O" is rare. The bar $\mid$ is what lets the model use the history, and teacher forcing is a fight over what history to condition on.

3. Ground truth vs. prediction — the hat "^"

The single most important distinction on the whole page:

The picture: two rows of boxes.

• Top row — the answer key, filled in ink (ground truth $y_t$).
• Bottom row — the model's attempt, filled in pencil (predictions ${\hat{y}}_t$), which may have eraser marks where it went wrong.

Why the topic needs it. Teacher forcing is exactly the choice: when the model needs a "previous token" to condition on, do we hand it the ink row ($y_{t-1}$, teacher forcing) or the pencil row (${\hat{y}}_{t-1}$, free running)? Without the hat you cannot even state the question.

4. Autoregressive — the pencil feeds itself

The picture: an arrow that loops from the bottom-row box at position $t-1$ back up into the model to produce position $t$. The pencil literally feeds itself.

Why it matters. Because of this self-feeding loop, a single wrong pencil mark at position $5$ becomes the input at position $6$, which corrupts position $6$, and so on — errors compound. Teacher forcing breaks the loop during training by feeding the ink row instead. See 4.2.3-Autoregressive-models for the general family.

5. The product over time — the ∏ symbol

To score a whole sentence, we multiply the per-position guesses:

$p(y\mid x)={\prod }_{t=1}^{T}p(y_t\mid y_{1:t-1},x)$

New notation, defined:

• ${\prod }_{t=1}^T$ — the big "pi" means multiply together, one factor for each $t$ from $1$ to $T$. (It is to multiplication what $\sum$ is to addition.)
• $x$ — the input / condition (e.g. the French sentence we're translating from). It sits inside every factor because every guess may depend on it.

Why the topic needs it. This product is the quantity we are trying to make large — a good model makes the true sentence highly probable. Teacher forcing is a recipe for how to compute each factor while training.

6. Turning products into sums — the log

Multiplying hundreds of tiny probabilities gives a ridiculously small number (like $0.9^{500}$), which computers round to zero. The fix is the logarithm.

Apply $\log$ to the product and it becomes a sum:

$\mathcal{L}=\log p(y\mid x)={\sum }_{t=1}^T\log p_{\theta }(y_t\mid y_{1:t-1},x)$

Newly-earned symbols:

• ${\sum }_{t=1}^T$ — "sigma", add up one term per position.
• $\mathcal{L}$ — the objective (script L), the number we want to steer.
• $\theta$ — "theta", the bundle of all the model's tunable weights. Writing $p_{\theta }$ means "the probability as computed by the model with settings $\theta$."

Why the topic needs it. Every "Loss" column in the parent note's worked tables is one $-\log p$ value. Understanding that curve — flat-and-tiny near $p=1$, exploding near $p=0$ — is why a cascade of wrong tokens (small $p$'s) makes the loss "explode" in the free-running example.

7. The teacher-forcing ratio — ε

• $\in$ — "is a member of"; $[0,1]$ — the closed range from $0$ to $1$ inclusive.
• The picture: a dial from $0$ to $1$. Turn it up for stable training, ease it down so the model learns to survive its own mistakes — that easing is 3.5.14-Scheduled-sampling, a cousin of 2.4.5-Curriculum-learning.

How the foundations feed the topic

The left branch builds the objective ($\mathcal{L}$); the right branch builds the input-choice (ink vs pencil). Teacher forcing sits where they meet, and the exposure-bias gap (5.3.8-Exposure-bias) is what happens when the training input-choice differs from inference.

Equipment checklist

Cover the right side; can you answer each before revealing?

When every line above is easy, you're ready for the full Teacher Forcing derivation and the 3.6.2-Seq2seq-architecture it lives inside.