Exercises — Teacher forcing

Training with teacher forcing: (b) the ground-truth $y_{t-1}$. The correct earlier token is handed in, like training wheels. Inference: it has no choice — the true continuation does not exist yet, so it must feed its own ${\hat{y}}_{t-1}$. This mismatch between the two situations is exactly exposure bias.

• $ϵ=1.0$: pure teacher forcing — always feed ground truth. Standard, fast, stable.
• $ϵ=0.0$: pure free running — always feed the model's own prediction. Hard, unstable.
• $ϵ=0.5$: scheduled sampling — flip a fair coin each step; sometimes truth, sometimes prediction. See 3.5.14-Scheduled-sampling. $ϵ$ is literally the probability of using the ground truth at any given step.

${\ell }_t=-\log (0.8)\approx 0.223$. Higher confidence on the right token would push this toward $0$.

Per-step ${\ell }_t=-\log p_t$: $-\log 0.7=0.357,-\log 0.8=0.223,-\log 0.9=0.105,$ $-\log 0.85=0.163,-\log 0.95=0.051,-\log 0.9=\mathrm{0.105.}$ Sum: $0.357+0.223+0.105+0.163+0.051+0.105=1.004$. Because every step got the true previous token, no error could cascade — each probability is read directly off a clean history.

$-\log 0.7=0.357,-\log 0.8=0.223,-\log 0.6=0.511,$ $-\log 0.35=1.050,-\log 0.2=1.609,-\log 0.5=\mathrm{0.693.}$ Sum $=0.357+0.223+0.511+1.050+1.609+0.693=4.443$. That is about 4.4× the teacher-forced loss. The spike at $t=4,5$ is the cascade: one wrong argmax became the next step's input, dragging the model into a state it never trained on. This is the numeric fingerprint of exposure bias.

• $k=30$: $1.0-0.30=0.70$, and $\max (0.5,0.70)=0.70$.
• $k=60$: $1.0-0.60=0.40$, but $\max (0.5,0.40)=0.50$ (floor kicks in).
• Floor reached when $1.0-0.01k=0.5\Rightarrow k=50$. For all $k\ge 50$, $ϵ=0.5$.

The culprit is $\arg \max$. It returns the index of the largest logit — a step function that is flat almost everywhere (tiny changes in logits don't change the chosen index) and jumps discontinuously when the top-2 tokens swap. A flat function has derivative $0$; a jump has no derivative. So $\partial {\hat{y}}_{t-1}/\partial \theta =0$ (or undefined), and the error signal at step $t$ cannot propagate through the previous-step selection. Teacher forcing fix: the input $y_{t-1}$ is a constant from the dataset, not a function of $\theta$. There is no $\arg \max$ in the path, so gradients from every step flow straight back to the parameters without passing through a non-differentiable choice. This is a large part of why teacher forcing trains fast and stably.

Each step independently uses prediction with probability $1-ϵ=0.5$. Expected prediction-steps $=T(1-ϵ)=6\times 0.5=3$. Probability all six use ground truth $={ϵ}^T=0.5^6=1/64\approx 0.0156$. So even at $ϵ=0.5$ a fully teacher-forced sequence is rare — the model spends most runs practising recovery, which is the whole point of the compromise.

Exposure bias is the gap between the loss on ground-truth histories and the loss on the model's own histories: $\Delta ={\mathcal{L}}_{\text{inference}}-{\mathcal{L}}_{TF}.$

• Student A: ${\Delta }_A=0.6-0.5=0.1$ (small — robust to its own mistakes).
• Student B: ${\Delta }_B=2.8-0.5=2.3$ (large — collapses once it feeds itself). Student B suffers far more. Equal training loss says nothing about robustness; only the gap does.

Linear part must go from $(20,1.0)$ to $(150,0.3)$. Slope $=(0.3-1.0)/(150-20)=-0.7/130\approx -0.005385$ per epoch. So:

• $k=10$: warm-up region $\Rightarrow 1.0$.
• $k=85$: $1.0-0.005385(65)=1.0-0.350=0.650$.
• $k=180$: floor region $\Rightarrow 0.30$. This mirrors curriculum learning: easy (stable) inputs first, harder (self-generated) inputs later.

h = h0
total_loss = 0
input_t = <START>            # first step: fixed start token, never a prediction
for t = 1 to T:
    output_t, h = f_theta(input_t, h)
    total_loss += CrossEntropy(output_t, y_t)
    # choose next input
    if t < T:
        if random() < epsilon:
            input_t = y_t          # ground truth (teacher forcing)
        else:
            input_t = argmax(output_t)   # model's own prediction (free running)
backprop(total_loss); update(theta)

Key points: the loop always scores against the true target $y_t$ (loss uses ground truth); only the next input is chosen by the coin flip. Step $1$ is special — it uses the fixed <START> token, so there is never an undefined "previous prediction." If $ϵ=1.0$: random() < 1.0 is always true, so input_t = y_t every step — the argmax branch is dead code. This recovers pure teacher forcing, and the loss reduces to ${\mathcal{L}}_{TF}={\sum }_t{\ell }_t$ of L2.1.

Teacher forcing lives only in the decoder, because only the decoder is autoregressive — its input at step $t$ is the previous output token $y_{t-1}$ (or ${\hat{y}}_{t-1}$). That previous token is where we choose truth vs prediction. The encoder reads the source sentence, which is fully known and fixed at both training and inference; it has no "previous prediction" to condition on, so the teacher-forcing knob simply doesn't apply to it. The context vector $c$ is the same either way; only how the decoder feeds itself changes.

Training only minimises ${\mathcal{L}}_{TF}$: the parameters are tuned so that ground-truth histories give low loss. The model has never optimised for its own-generated histories. Whenever a self-generated token differs from the truth, the model enters a context it was not trained on, so on average it assigns less probability to the correct next token there $\Rightarrow$ higher per-step loss $\Rightarrow \Delta \ge 0$ typically. When negative? Rarely, and only by luck: if the model's own predictions happen to exactly match the ground truth for a particular sequence, then the two histories are identical and $\Delta =0$. And on an individual noisy example the self-history could accidentally lead to a confidently-correct region, giving a slightly negative gap — but this does not survive averaging over data, because training never rewarded off-truth states. So the expected gap is $\ge 0$.

(A) $-\log 0.9-\log 0.85-\log 0.95=0.1054+0.1625+0.0513=0.319$. (B) $-\log 0.9-\log 0.80-\log 0.90=0.1054+0.2231+0.1054=0.434$. (C) $-\log 0.9-\log 0.30-\log 0.15=0.1054+1.2040+1.8971=3.206$. Ranking (best $\to$ worst loss): $\text{A}<\text{B}<\text{C}$. Reading: teacher forcing is cheapest but least like inference; free running with a cascade explodes; scheduled sampling pays a modest premium ($0.434$ vs $0.319$) to buy robustness — the sweet spot.

Idea — "soft-mix teacher forcing": instead of feeding either the hard truth token or a hard argmax, feed a soft blend of the truth embedding and the model's softmax-weighted predicted embedding: ${\text{input}}_t=\alpha e(y_{t-1})+(1-\alpha ){\sum }_v{p}_{\theta }(v)e(v),$ where $e(\cdot )$ is the token embedding and $p_{\theta }(v)$ is the model's full probability distribution over vocabulary $v$ (not an argmax). Why it dodges the trap: the second term is a smooth, differentiable function of $\theta$ (a weighted average of embeddings, weights = softmax probabilities). No argmax anywhere, so gradients flow cleanly — unlike hard free running. Why it shrinks exposure bias: as $\alpha \to 0$ the model conditions on its own belief, so it practises recovering from its own imperfect states, narrowing the train/test gap. Annealing $\alpha$ from $1\to$ small over epochs mirrors scheduled sampling but stays fully differentiable. Trade-off honestly stated: the soft blend is not exactly the discrete token the model will see at inference, so a residual gap remains — no free lunch, but a strictly better gradient than straight-through argmax.

Teacher forcing feeds what during training? ::: The ground-truth previous token $y_{t-1}$. The single operation that blocks gradients in free running? ::: $\arg \max$ (flat/discontinuous). Exposure bias equals which difference? ::: ${\mathcal{L}}_{\text{inference}}-{\mathcal{L}}_{TF}$. $ϵ=1.0$ means? ::: Pure teacher forcing, every step. Expected prediction-steps at $ϵ=0.5$ over $T=6$? ::: $3$.