Teacher forcing

3.5.13 · AI-ML › Sequence Models

Yeh bilkul bike sikhne jaisa hai — training wheels ke saath versus bina training wheels ke. Training wheels ke saath (teacher forcing), tumhe har moment stable support milta hai. Unke bina, ek wobble agle wobble ko compound karta hai. Teacher forcing sequence models ke liye training wheels hai.

Why This Matters

Problem: Sequence models autoregressive hote hain — har prediction pichli predictions par depend karti hai. Inference ke dauran, hum apne outputs ko next step ke inputs ke roop mein use karne par majboor hote hain. Lekin training ke dauran, hamare paas correct answer (training data) hota hai. Isse ek fundamental sawaal uthta hai: training kaise karein.

Do Approaches:

1. Teacher Forcing (ground truth use karna): Fast, stable training lekin exposure bias
2. Free Running (model predictions use karna): Inference se match karta hai lekin training slow aur unstable hoti hai

Formal Definition: Given sequence $y=(y_1,y_2,...,y_T)$, training time par:

• Teacher forcing ke saath: $p(y_t|y_{1:t-1},x)\approx p_{\theta }(y_t|y_{1:t-1},x)$ jahan $y_{1:t-1}$ ground truth hai
• Teacher forcing ke bina: $p(y_t|{\hat{y}}_{1:t-1},x)\approx p_{\theta }(y_t|{\hat{y}}_{1:t-1},x)$ jahan ${\hat{y}}_{1:t-1}$ model predictions hain

Derivation: Teacher Forcing Kyon Kaam Karta Hai

Sequence Prediction Objective

Chalte hain first principles se derive karte hain. Hum chahte hain input $x$ given sequence $y=(y_1,..,y_T)$ ki probability model karein:

$p(y|x)={\prod }_{t=1}^{T}p(y_t|y_{1:t-1},x)$

Yeh factorization kyun? Probability ka chain rule: $p(A,B,C)=p(A)p(B|A)p(C|A,B)$. Hum isse sirf sequences par apply kar rahe hain.

Training Objective log-likelihood maximize karna hai:

$\mathcal{L}=\log p(y|x)={\sum }_{t=1}^T\log p_{\theta }(y_t|y_{1:t-1},x)$

Log kyun? Kyunki products sums ban jaate hain (easier optimization), aur yeh negative cross-entropy hai (hamara actual loss function).

Ise Compute Karne Ke Do Tarike

Timestep $t$ par, hume $p_{\theta }(y_t|y_{1:t-1},x)$ compute karni hai. Lekin $y_{1:t-1}$ kya hai?

Option 1: Teacher Forcing $y_{1:t-1}=\text{ground truth from training data}$

Yeh kyun help karta hai: Har timestep ko correct context milta hai. Model sikhta hai: "Sahi history given ho, toh next token predict karo." Gradients cleanly flow karte hain kyunki hum previous predictions ke through backpropagate nahi kar rahe.

Option 2: Free Running ${\hat{y}}_{1:t-1}=\text{argmax of model’s previous outputs}$

Yeh kyun mushkil hai: Agar $t=5$ par galti ho, toh $t=6$ par hum ek galat input par conditioning kar rahe hain. Error compound hota hai. Gradients ko puri sequence of previous argmax operations ke through flow karna padta hai (jo non-differentiable hai!).

Mathematical Trade-off

Teacher Forcing Loss (jo hum actually optimize karte hain): ${\mathcal{L}}_{TF}={\sum }_{t=1}^T\log p_{\theta }(y_t|y_{1:t-1},x)$

Inference Reality (jo hum actually karte hain): ${\mathcal{L}}_{inference}={\sum }_{t=1}^T\log p_{\theta }(y_t|{\hat{y}}_{1:t-1},x)$

Yeh alag distributions hain! Exposure bias inke beech ka gap hai.

Mathematically yeh kyun matter karta hai: Model training ke dauran kabhi apni galtiyan nahi dekhta. Test time par, ek error cascade kar sakti hai kyunki model ek aisi state mein hota hai (apni galat prediction receive karte hue) jiske liye usne kabhi train nahi kiya.

Input:

• Model $f_{\theta }$ with parameters $\theta$
• Training sequence $(x,y)$ jahan $y=(y_1,...,y_T)$
• Teacher forcing ratio $ϵ\in [0,1]$

Har training step ke liye:

initialize hidden state h₀
total_loss = 0

for t = 1 to T:
    if random() < ε:  // Teacher forcing
        input_t = y_{t-1}  // Use ground truth
    else:              // Free running
        input_t = argmax(output_{t-1})  // Use prediction
    
    output_t, h_t = f_θ(input_t, h_{t-1})
    loss_t = CrossEntropy(output_t, y_t)
    total_loss += loss_t

backpropagate(total_loss)
update(θ)

Har component kyun:

• $ϵ=1.0$: Pure teacher forcing (standard)
• $ϵ=0.0$: Pure free running (train karna mushkil)
• $0<ϵ<1$: Scheduled sampling (compromise)

Worked Examples

Task: RNN ko "HELLO" character by character generate karna sikhao.

Setup:

• Vocabulary: {H, E, L, O, <START>, <END>}
• Input sequence: <START> H E L L O
• Target sequence: H E L L O <END>

Teacher Forcing ke saath Training (ε=1.0):

Yeh step kyun? t=2 par, chahe model ne t=1 par H sirf 70% confidence se predict kiya, hum phir bhi usse true 'H' feed karte hain t=2 par. Isse model track par rehta hai.

Total Loss: $0.36+0.22+0.11+0.16+0.05=0.90$

Teacher Forcing ke bina Training (ε=0.0):

Ruko! t=3 par model kam confident hai (0.6 vs 0.9) kyunki t=2 par usne apni khud ki prediction H dekhi (jo thodi galat thi). Ab maan lo yeh argmax se L predict karta hai, lekin:

| t=4 | L (pred from t=3) | [O:0.4, L:0.35, E:0.25] | L | $-\log (0.35)=1.05$ |

Yeh step kyun? t=4 par, model galat O predict karta hai (argmax=0.4) jab usse L predict karna chahiye tha. Ab t=5 par:

| t=5 | O (wrong pred!) | [E:0.5, L:0.3, O:0.2] | O | $-\log (0.2)=1.61$ |

Model confused hai kyunki woh kabhi is state mein nahi raha (abhi O output karke). Loss explode ho jaata hai: $0.36+0.22+0.51+1.05+1.61=3.75$ (4× worse!).

Key Insight: Teacher forcing ke bina, ek error cascade karta hai. Teacher forcing ke saath, har step ko clean slate milti hai.

Compromise: $ϵ=1.0$ (pure teacher forcing) se shuru karo, phir gradually decay karo:

${ϵ}_k=\max (0.5,1.0-k\cdot 0.01)$

jahan $k$ epoch number hai.

Yeh kyun kaam karta hai:

• Early training: Model basic patterns sikhta hai stable inputs ke saath ($ϵ\approx 1$)
• Late training: Model apni galtiyon se recover karna sikhta hai ($ϵ\approx 0.5$)

Epoch 1 ($ϵ=1.0$):

Input:  [<START>, H,     E,     L,     L    ]  ← sab ground truth
Output: [H,       E,     L,     L,     O    ]
Loss:   [0.36,    0.22,  0.11,  0.16,  0.05 ] = 0.90

Epoch 50 ($ϵ=0.5$):

Step 1: input=<START> → predict H (use truth: H)
Step 2: input=H       → predict E (flip coin: use prediction E)
Step 3: input=E       → predict L (flip coin: use truth L)
Step 4: input=L       → predict L (flip coin: use prediction L)
Step 5: input=L       → predict O

Model ab mix of perfect aur imperfect histories par train karta hai, train-test gap ko bridge karta hua.

Task: "Je suis" → "I am" translate karo

Encoder-Decoder with Teacher Forcing:

Encoder: [Je, suis] → context vector c

Decoder Training (teacher forcing ke saath):
t=1: input =<START>, context = c
     → output = [I:0.9, You:0.05, ...] 
     → target = I
     → loss = -log(0.9) = 0.11

t=2: input = I (ground truth!), context = c
     → output = [am:0.85, are:0.1...]
     → target = am
     → loss = -log(0.85) = 0.16

t=3: input = am (ground truth!), context = c
     → output = [<END>:0.95, ...]
     → target = <END>
     → loss = -log(0.95) = 0.05

Yeh step kyun (t=2)? Chahe model ne t=1 par "I" par sirf 90% confidence rakhi, hum usse correct "I" t=2 par feed karte hain. Isse decoder "You is" ya aur koi nonsense mein spiral karne se bachta hai.

Inference par (koi teacher nahi, free-run karna padega):

t=1: input = <START> → output = I (confident)
t=2: input = I (apni prediction) → output = am (abhi bhi theek)
t=3: input = am → output = <END>
Result: "I am" ✓

Lekin agar t=1 ne "You" galti se predict kiya hota:

t=1: input = <START> → output = You (galti!)
t=2: input = You (wrong!) → output = are (model ne "You are" saath seekha tha)
Result: "You are" ✗

Yahi exposure bias hai: model ne kabhi is state par train nahi kiya — "main abhi galti se 'You' bol chuka hoon, ab kya karoon?"

Common Mistakes

Yeh sahi kyun lagta hai: Agar hum model ko hamesha correct answer feed karte hain, toh kya woh sirf copy nahi kar raha?

Yeh galat kyun hai: Model next token nahi dekhta; woh sirf previous tokens ko input ke roop mein dekhta hai. Har step par, use abhi bhi context given next token predict karna hota hai. Teacher forcing correct context provide karta hai, answer nahi.

Example: "HELLO" mein t=3 par, hum model ko "HE" (correct context) feed karte hain, lekin usse abhi bhi "L" output karna seekhna hai. Model ko nahi bataya jaata "answer L hai" — use loss signal se H→E→L pattern seekhna hota hai.

Fix: Samjho ki teacher forcing inputs ko stabilize karta hai lekin model phir bhi prediction errors ke backpropagation se outputs sikhta hai.

Yeh sahi kyun lagta hai: Agar teacher forcing kaam karta hai, toh ise 100% time kyun na use karein?

Yeh galat kyun hai: Pure teacher forcing exposure bias create karta hai. Model training ke dauran kabhi apni galtiyan experience nahi karta, isliye test time par unse recover nahi kar sakta.

Example: $ϵ=1.0$ se trained ek chatbot:

• Training: "How are you?" → "I am fine" (perfect, har baar)
• Test: "How are you?" → "I ma fine" (typo) → "fine you are how" (spiral!)

Model ne kabhi nahi seekha typo hone par kya karna hai, isliye error compound ho jaata hai.

Fix: Scheduled sampling use karo. Stable early training ke liye $ϵ=1.0$ se shuru karo, phir $ϵ=0.5$ ya 0.0 tak decay karo. Model dono correct patterns AUR error recovery sikhta hai.

Yeh sahi kyun lagta hai: Koi bhi sequence model ek baar mein ek token predict karta hai, hai na?

Yeh galat kyun hai: Teacher forcing sirf autoregressive models par apply hota hai (jahan model ka output uska input ban jaata hai). Yeh apply nahi hota:

• Encoder-only models (BERT): Koi generation nahi, isliye koi sequential dependencies nahi
• Non-autoregressive models (parallel generation): Saare tokens simultaneously predict hote hain
• Sequence labeling (POS tagging): Input aur output same length ke hain aur independent hain

Example: BERT mein, hum "The cat sat on the ___" mask karte hain, aur "mat" predict karte hain. Koi previous token nahi hai input ke roop mein feed karne ke liye — hum masked prediction kar rahe hain, sequential generation nahi.

Fix: Samjho ki teacher forcing specifically autoregressive generation ke liye hai (GPT, RNN language models, seq2seq decoders).

Practical Considerations

Teacher Forcing kab use karein:

• ✓ RNN/LSTM/GRU language models
• ✓ Seq2seq decoders (translation, summarization)
• ✓ Autoregressive Transformers (GPT-style)
• ✓ Early training stages (hamesha stable)

Kab reduce karein ya avoid karein:

• ✗ Late training (khud ki galtiyan dekhni chahiye)
• ✗ Tasks jahan errors badly compound hoti hain (long-form generation)
• ✗ Jab tumhare paas curriculum learning strategy ho

Scheduled Sampling Strategies:

1. Linear decay: ${ϵ}_k=\max ({ϵ}_{min},1.0-k/K)$
2. Exponential decay: ${ϵ}_k={ϵ}_{min}+(1-{ϵ}_{min})\cdot e^{-k/\tau }$
3. Inverse sigmoid: ${ϵ}_k=\frac{k}{k+e^{k/\tau }}$

Decay kyun matter karta hai: Yeh curriculum learning ka ek form hai. Easy se shuru karo (stable inputs), hard ki taraf badho (noisy inputs jo test time se match karein).

Teacher forcing aisa hai jaise tumhara teacher tumhe correct previous words batata rahe jab tum naya word likhte ho. Chahe tumne "Once upan" likh diya (oops, typo!), teacher usse "upon" se correct karta hai agle word se pehle. Isse tumhe sahi patterns seekhne mein madad milti hai bina apni galtiyon se confuse hue.

Teacher forcing ke bina aisa hai jaise poori kahani akele likhna. Agar tum "upan" galat likhte ho, toh tum confuse ho sakte ho aur "upan the tiem" likh sakte ho — tumhari galtiyan stack hoti jaati hain!

Smart tarika (scheduled sampling) yeh hai: Pehle, teacher ko bahut help karne do (taaki tum basic story flow seekho). Phir gradually khud likhne do (taaki tum apni galtiyan pakadna seekho). Aakhir mein, tum bina help ke poori kahaniyan likh sakte ho!

Bilkul aise hi hum AI ko sentences likhna train karte hain: training wheels (teacher) se shuru karo, phir dheere dheere hata do taaki AI khud likh sake.

Teacher training ke dauran tumhe track par rehne ke liye force karta hai, lekin test time par tum akele ho — isliye tumhe dheere dheere apni galtiyan handle karna seekhna hoga (time ke saath epsilon reduce karo).

Connections

• 3.5.1-RNN-fundamentals: Teacher forcing RNN training ko stabilize karta hai
• 3.5.7-LSTM: LSTMs ko abhi bhi autoregressive tasks ke liye teacher forcing chahiye
• 3.5.14-Scheduled-sampling: Exposure bias ka solution
• 3.6.2-Seq2seq-architecture: Decoder training teacher forcing use karta hai
• 4.2.3-Autoregressive-models: Kisi bhi autoregressive training ke liye core concept
• 5.3.8-Exposure-bias: Woh fundamental problem jo teacher forcing create karta hai
• 2.4.5-Curriculum-learning: Scheduled sampling as curriculum

#flashcards/ai-ml

What is teacher forcing? :: Ek training technique jisme har timestep par, hum model ko training data se true previous token as input feed karte hain, rather than model ki apni prediction ke.

What is scheduled sampling? :: Training ke dauran teacher forcing ratio ko gradually decrease karna, ε ≈ 1.0 (stable early training) se shuru karke ε ≈ 0.5 ya lower (error recovery seekhne ke liye) tak decay karna.

Concept Map