Teacher forcing

Think of it like learning to ride a bike with training wheels versus without. With training wheels (teacher forcing), you get stable support at every moment. Without them, one woble compounds into the next. Teacher forcing is the training wheels for sequence models.

Why This Matters

The Problem: Sequence models are autoregressive—each prediction depends on previous predictions. During inference, we have no choice but to use our own outputs as inputs for the next step. But during training, we do have the correct answer (the training data). This creates a fundamental question about how to train.

Two Approaches:

1. Teacher Forcing (using ground truth): Fast, stable training but exposure bias
2. Free Running (using model predictions): Matches inference but slow, unstable training

Formal Definition: Given sequence $y = (y_1, y_2, ..., y_T)$, at training time:

• With teacher forcing: $p(y_t | y_{1:t-1}, x) \approx p_\theta(y_t | y_{1:t-1}, x)$ where $y_{1:t-1}$ are ground truth
• Without teacher forcing: $p(y_t | \hat{y}_{1:t-1}, x) \approx p_\theta(y_t | \hat{y}_{1:t-1}, x)$ where $\hat{y}_{1:t-1}$ are model predictions

Derivation: Why Teacher Forcing Works

The Sequence Prediction Objective

Let's derive this from first principles. We want to model the probability of a sequence $y = (y_1, .., y_T)$ given input $x$:

$p(y|x) = \prod_{t=1}^{T} p(y_t | y_{1:t-1}, x)$

Why this factorization? Chain rule of probability: $p(A,B,C) = p(A)p(B|A)p(C|A,B)$. We're just applying it to sequences.

The Training Objective is to maximize log-likelihood:

$\mathcal{L} = \log p(y|x) = \sum_{t=1}^{T} \log p_\theta(y_t | y_{1:t-1}, x)$

Why log? Because products become sums (easier optimization), and it's the negative cross-entropy (our actual loss function).

Two Ways to Compute This

At timestep $t$, we need to compute $p_\theta(y_t | y_{1:t-1}, x)$. But what is $y_{1:t-1}$?

Option 1: Teacher Forcing $y_{1:t-1} = \text{ground truth from training data}$

Why this helps: Every timestep gets the correct context. The model learns: "Given the right history, predict the next token." Gradients flow cleanly because we're not backpropagating through previous predictions.

Option 2: Free Running $\hat{y}_{1:t-1} = \text{argmax of model's previous outputs}$

Why this is hard: If we make a mistake at $t=5$, then at $t=6$ we're conditioning on a wrong input. The error compounds. Gradients must flow through the entire sequence of previous argmax operations (non-differentiable!).

The Mathematical Trade-off

Teacher Forcing Loss (what we actually optimize): $\mathcal{L}_{TF} = \sum_{t=1}^{T} \log p_\theta(y_t | y_{1:t-1}, x)$

Inference Reality (what we actually do): $\mathcal{L}_{inference} = \sum_{t=1}^{T} \log p_\theta(y_t | \hat{y}_{1:t-1}, x)$

These are different distributions! Exposure bias is the gap between them.

Why this matters mathematically: The model never sees its own mistakes during training. At test time, one error can cascade because the model is in a state (receiving its own wrong prediction) it never trained for.

Input:

• Model $f_\theta$ with parameters $\theta$
• Training sequence $(x, y)$ where $y = (y_1, ..., y_T)$
• Teacher forcing ratio $\epsilon \in [0, 1]$

For each training step:

initialize hidden state h₀
total_loss = 0

for t = 1 to T:
    if random() < ε:  // Teacher forcing
        input_t = y_{t-1}  // Use ground truth
    else:              // Free running
        input_t = argmax(output_{t-1})  // Use prediction
    
    output_t, h_t = f_θ(input_t, h_{t-1})
    loss_t = CrossEntropy(output_t, y_t)
    total_loss += loss_t

backpropagate(total_loss)
update(θ)

Why each component:

• $\epsilon = 1.0$: Pure teacher forcing (standard)
• $\epsilon = 0.0$: Pure free running (hard to train)
• $0 < \epsilon < 1$: Scheduled sampling (compromise)

Worked Examples

Task: Train RNN to generate "HELLO" character by character.

Setup:

• Vocabulary: {H, E, L, O , }
• Input sequence: <START> H E L L O
• Target sequence: H E L L O <END>

Training with Teacher Forcing (ε=1.0):

Why this step? At t=2, even though the model predicted H with only 70% confidence at t=1, we still feed it the true 'H' at t=2. This keeps the model on track.

Total Loss: $0.36 + 0.22 + 0.11 + 0.16 + 0.05 = 0.90$

Training without Teacher Forcing (ε=0.0):

Wait! At t=3, the model is less confident (0.6vs 0.9) because at t=2 it saw its own prediction of H (which was slightly wrong). Now let's say it predicts L with argmax, but:

| t=4 | L (pred from t=3) | [O:0.4, L:0.35, E:0.25] | L | $-\log(0.35) = 1.05$ |

Why this step? At t=4, the model wrongly predicts O (argmax=0.4) when it should predict L. Now at t=5:

| t=5 | O (wrong pred!) | [E:0.5, L:0.3, O:0.2] | O | $-\log(0.2) = 1.61$ |

The model is confused because it's never been in this state before (having just output O). The loss explodes: $0.36 + 0.22 + 0.51 + 1.05 + 1.61 = 3.75$ (4× worse!).

Key Insight: Without teacher forcing, one error cascades. With teacher forcing, each step gets a clean slate.

The Compromise: Start with $\epsilon = 1.0$ (pure teacher forcing), then gradually decay:

$\epsilon_k = \max(0.5, 1.0 - k \cdot 0.01)$

where $k$ is the epoch number.

Why this works:

• Early training: Model learns basic patterns with stable inputs ($\epsilon \approx 1$)
• Late training: Model learns to recover from its own errors ($\epsilon \approx 0.5$)

Epoch 1 ($\epsilon = 1.0$):

Input:  [<START>, H,     E,     L,     L    ]  ← all ground truth
Output: [H,       E,     L,     L,     O    ]
Loss:   [0.36,    0.22,  0.11,  0.16,  0.05 ] = 0.90

Epoch 50 ($\epsilon = 0.5$):

Step 1: input=<START> → predict H (use truth: H)
Step 2: input=H       → predict E (flip coin: use prediction E)
Step 3: input=E       → predict L (flip coin: use truth L)
Step 4: input=L       → predict L (flip coin: use prediction L)
Step 5: input=L       → predict O

The model now trains on a mix of perfect and imperfect histories, bridging the train-test gap.

Task: Translate "Je suis" → "I am"

Encoder-Decoder with Teacher Forcing:

Encoder: [Je, suis] → context vector c

Decoder Training (with teacher forcing):
t=1: input =<START>, context = c
     → output = [I:0.9, You:0.05, ...] 
     → target = I
     → loss = -log(0.9) = 0.11

t=2: input = I (ground truth!), context = c
     → output = [am:0.85, are:0.1...]
     → target = am
     → loss = -log(0.85) = 0.16

t=3: input = am (ground truth!), context = c
     → output = [<END>:0.95, ...]
     → target = <END>
     → loss = -log(0.95) = 0.05

Why this step (t=2)? Even if the model only had 90% confidence on "I" at t=1, we feed it the correct "I" at t=2. This prevents the decoder from spiraling into "You is" or other nonsense.

At Inference (no teacher, must free-run):

t=1: input = <START> → output = I (confident)
t=2: input = I (our own prediction) → output = am (still okay)
t=3: input = am → output = <END>
Result: "I am" ✓

But if t=1 had predicted "You" by mistake:

t=1: input = <START> → output = You (mistake!)
t=2: input = You (wrong!) → output = are (the model learned "You are" together)
Result: "You are" ✗

This is exposure bias: the model never trained on the state "I just wrongly said 'You', now what?"

Common Mistakes

Why this feels right: If we always feed the model the correct answer, isn't it just copying?

Why it's wrong: The model doesn't see the next token; it only sees previous tokens as input. At each step, it still has to predict the next token given the context. Teacher forcing provides the correct context, not the answer.

Example: At t=3 in "HELLO", we feed the model "HE" (correct context), but it must still learn to output "L". The model isn't told "the answer is L"—it must learn the pattern H→E→L from the loss signal.

The fix: Understand that teacher forcing stabilizes inputs but the model still learns outputs through backpropagation of prediction errors.

Why this feels right: If teacher forcing works, why not use it 100% of the time?

Why it's wrong: Pure teacher forcing creates exposure bias. The model never experiences its own errors during training, so it can't learn to recover from them at test time.

Example: A chatbot trained with $\epsilon = 1.0$ might:

• Training: "How are you?" → "I am fine" (perfect, every time)
• Test: "How are you?" → "I ma fine" (typo) → "fine you are how" (spiral!)

The model never learned what to do when it makes a typo, so it compounds the error.

The fix: Use scheduled sampling. Start with $\epsilon = 1.0$ for stable early training, then decay to $\epsilon = 0.5$ or even 0.0. The model learns both correct patterns AND error recovery.

Why this feels right: Any sequence model predicts one token at a time, right?

Why it's wrong: Teacher forcing only applies to autoregressive models (where the model's output becomes its input). It doesn't apply to:

• Encoder-only models (BERT): No generation, so no sequential dependencies
• Non-autoregressive models (parallel generation): All tokens predicted simultaneously
• Sequence labeling (POS tagging): Input and output are the same length and independent

Example: In BERT, we mask "The cat sat on the ___", and predict "mat". There's no previous token to feed as input—we're doing masked prediction, not sequential generation.

The fix: Understand that teacher forcing is specifically for autoregressive generation (GPT, RNN language models, seq2seq decoders).

Practical Considerations

When to use Teacher Forcing:

• ✓ RNN/LSTM/GRU language models
• ✓ Seq2seq decoders (translation, summarization)
• ✓ Autoregressive Transformers (GPT-style)
• ✓ Early training stages (always stable)

When to reduce or avoid:

• ✗ Late training (need to see own errors)
• ✗ Tasks where errors compound badly (long-form generation)
• ✗ When you have a curriculum learning strategy

Scheduled Sampling Strategies:

1. Linear decay: $\epsilon_k = \max(\epsilon_{min}, 1.0 - k/K)$
2. Exponential decay: $\epsilon_k = \epsilon_{min} + (1 - \epsilon_{min}) \cdot e^{-k/\tau}$
3. Inverse sigmoid: $\epsilon_k = \frac{k}{k + e^{k/\tau}}$

Why decay matters: It's a form of curriculum learning. Start easy (stable inputs), progress to hard (noisy inputs that match test time).

Teacher forcing is like having your teacher tell you the correct previous words as you write each new word. Even if you wrote "Once upan" (ops, typo!), the teacher corrects it to "upon" before you write the next word. This helps you learn the right patterns without getting confused by your own mistakes.

Without teacher forcing is like writing the whole story alone. If you misspell "upan," then you might get confused and write "upan the tiem"—your mistakes pile up!

The smart way (scheduled sampling) is: First, let the teacher help a lot (so you learn the basic story flow). Then gradually let you write more on your own (so you learn to catch your own mistakes). By the end, you can write full stories without help!

That's exactly how we train AI to write sentences: start with training wheels (teacher), then gradually remove them so AI can write on its own.

The teacher forces you to stay on track during training, but at test time you're on your own—so you need to gradually learn to handle your own mistakes (reduce epsilon over time).

Connections

• 3.5.1-RNN-fundamentals: Teacher forcing stabilizes RNN training
• 3.5.7-LSTM: LSTMs still need teacher forcing for autoregressive tasks
• 3.5.14-Scheduled-sampling: The solution to exposure bias
• 3.6.2-Seq2seq-architecture: Decoder training uses teacher forcing
• 4.2.3-Autoregressive-models: Core concept for any autoregressive training
• 5.3.8-Exposure-bias: The fundamental problem teacher forcing creates
• 2.4.5-Curriculum-learning: Scheduled sampling as curriculum

#flashcards/ai-ml

What is teacher forcing? :: A training technique where at each timestep, we feed the model the true previous token from training data as input, rather than the model's own prediction.

What is scheduled sampling? :: Gradually decreasing the teacher forcing ratio during training, starting with ε ≈ 1.0 (stable early training) and decaying towardε ≈ 0.5 or lower (learning error recovery).

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)