Sequence-to-sequence models

3.5.7 · AI-ML › Sequence Models

What Problem Does Seq2Seq Solve?

Challenge yeh hai: Standard neural networks fixed input/output sizes expect karte hain. Lekin machine translation, summarization, chatbots mein, input "Bonjour" → output "Hello" ka 1 token → 1 token hai, jabki "Comment ça va?" → "How are you doing?" ka 3 → 4 tokens hai.

Breakthrough yeh hai: Input processing ko output generation se alag karo.

• Encoder: Poori input sequence $x_1,x_2,\dots ,x_{T_x}$ padhta hai aur ek context vector $\mathbf{c}$ produce karta hai (yeh "thought" hai).
• Decoder: Output sequence $y_1,y_2,\dots ,y_{T_y}$ generate karta hai $\mathbf{c}$ ke condition par, ek token at a time.

Yeh kyun kaam karta hai: Context vector ek learned, compressed representation hai. Decoder seekhta hai ki is representation ko target language/format mein "unfold" kaise karein.

Architecture Derivation from First Principles

Step 1: The Encoder

Hume ek variable-length input process karni hai. Sequences ko handle karne ke liye kaun sa structure use karte hain? RNNs, LSTMs, ya GRUs.

Maano encoder ek LSTM hai. Har time step $t$ par: $h_t^{(enc)}=\text{LSTM}(x_t,h_{t-1}^{(enc)})$

Saare $T_x$ tokens process karne ke baad, final hidden state $h_{T_x}^{(enc)}$ mein poori input ki information hoti hai. Hum set karte hain: $\mathbf{c}=h_{T_x}^{(enc)}$

Final state kyun? Ek LSTM mein, information forward flow karti hai. Last hidden state ne saare previous tokens "dekhe" hote hain (haalaanki yeh early waalon ko bhool sakta hai—yahi baad mein attention ko motivate karta hai).

Step 2: The Decoder

Decoder output sequence generate karta hai. Har decoding step $t$ par: $h_t^{(dec)}=\text{LSTM}(\mathbf{c},y_{t-1},h_{t-1}^{(dec)})$

Ruko, $y_{t-1}$ kya hai? Previous output token. Training ke dauran hum teacher forcing use karte hain: ground-truth previous token feed karte hain. Inference ke dauran, hum model ki apni prediction feed karte hain.

Decoder ka hidden state $h_t^{(dec)}$ ek softmax layer se guzara jaata hai next token predict karne ke liye: $P(y_t\mid y_{<t},\mathbf{c})=\text{softmax}(W_o{h}_t^{(dec)}+b_o)$

Softmax kyun? Hum vocabulary $V$ mein se ek token choose kar rahe hain, toh hume $|V|$ classes par ek probability distribution chahiye.

Poori sequence probability (yeh maanke ki $\mathbf{c}$ given conditional independence hai): $P(y_1,\dots ,y_{T_y}\mid x_1,\dots ,x_{T_x})={\prod }_{t=1}^{T_y}P(y_t\mid y_{<t},\mathbf{c})$

Step 3: Training with Cross-Entropy

Humare paas ground-truth pairs $(x,y)$ hain. Ek example ka loss: $\mathcal{L}=-{\sum }_{t=1}^{T_y}\log P(y_t^{\ast }\mid y_{<t}^{\ast },\mathbf{c})$

jahaan $y_t^{\ast }$ position $t$ par sahi target token hai.

Negative log-likelihood kyun? Hum correct sequence ki probability maximize karna chahte hain. $\log P(y\mid x)$ maximize karna $-\log P(y\mid x)$ minimize karne ke equivalent hai.

Backpropagation: Gradients decoder ke through backward flow karte hain, $\mathbf{c}$ mein jaate hain, phir encoder ke through. Yeh backpropagation through time (BPTT) hai do RNNs ke across.

Training vs. Inference: The Critical Difference

Training (Teacher Forcing)

Input:  "I love AI" 
Target: "<SOS> Me encanta IA <EOS>"

Decoder sees at t=1: <SOS> (from target)
Decoder predicts: "Me" (loss if wrong)

Decoder sees at t=2: "Me" (ground truth, not prediction!)
Decoder predicts: "encanta"
... and so on

Teacher forcing kyun? Training speed up hoti hai. Model ko har step par sahi context milta hai, instead of compounding errors ke.

Inference (Autoregressive Generation)

Input: "I love AI"
Decoder starts with <SOS>

t=1: Decoder predicts "Me" (from <SOS>)
t=2: Decoder uses "Me" to predict "encanta"
t=3: Decoder uses "encanta" to predict "IA"
t=4: Decoder predicts <EOS>, stop.

Exposure bias problem: Training mein, decoder kabhi apni galtiyan nahin dekhta. Inference mein, ek galat prediction cascade ho sakti hai. Yeh teacher forcing ki ek jaani-mani weakness hai.

Worked Example: English → Spanish Translation

Input: "I am happy"
Target: "Estoy feliz"

Encoding Phase

Vocabulary: {I, am, happy} → {0, 1, 2}
Embed: $x_1=E[0],x_2=E[1],x_3=E[2]$

Encoder LSTM (simplified, $d_h=4$):

h_0 = [0,0,0]
h_1 = LSTM(x_1, h_0) → [0.2, -0.1, 0.5, 0.3]
h_2 = LSTM(x_2, h_1) → [0.4, 0.2, 0.1, -0.2]
h_3 = LSTM(x_3, h_2) → [0.1, 0.6, -0.3, 0.4]

Context vector c = h_3 = [0.1, 0.6, -0.3, 0.4]

Decoding Phase

Target vocab: {, Estoy, feliz, } → {0, 1, 2, 3}

Decoder ko $\mathbf{c}$ se initialize karo:

Input: y_0 = <SOS> (token 0)
h_1^(dec) = LSTM(c, E[0], h_0^(dec))
Output: P(y_1|c) = softmax(W_o h_1^(dec))
       = [0.01, 0.92, 0.05, 0.02]  # Predicts "Estoy" (token 1)

Input: y_1 = "Estoy" (token 1, teacher forcing)
h_2^(dec) = LSTM(c, E[1], h_1^(dec))
Output: P(y_2|y_1,c) = [0.02, 0.03, 0.93, 0.02]  # Predicts "feliz" (token 2)

Input: y_2 = "feliz" (token 2)
h_3^(dec) = LSTM(c, E[2], h_2^(dec))
Output: P(y_3|y_1,y_2,c) = [0.01, 0.02, 0.03, 0.94]  # Predicts <EOS> (token 3)

Loss calculation (yeh maanke ki yeh sahi predictions hain): $\mathcal{L}=-\log (0.92)-\log (0.93)-\log (0.94)\approx 0.24$

Yeh step kyun? Hum correct tokens ke negative log-probs sum karte hain. Kam loss matlab correct sequence mein zyada confidence.

Common Decoding Strategies

1. Greedy Decoding

Har step par highest-probability token chuno: $y_t=\arg {\max }_{v}P(v\mid y_{<t},\mathbf{c})$

Fast hai par suboptimal: Future consequences consider nahin karta. Ab best word pick karna baad mein dead-end le ja sakta hai.

2. Beam Search

Har step par top-$k$ candidate sequences maintain karo. Expand karo, phir top-$k$ rakhho.

Beam width $k=2$ ke saath example:

Step 1: <SOS> → {"Estoy" (0.9), "Yo" (0.7)}

Step 2 (expand both):
  "Estoy" → {"Estoy feliz" (0.9×0.8=0.72), "Estoy contento" (0.9×0.6=0.54)}
  "Yo" → {"Yo soy" (0.7×0.5=0.35), "Yo estoy" (0.7×0.4=0.28)}
  
Keep top 2: {"Estoy feliz" (0.72), "Estoy contento" (0.54)}

Beam search kyun? Exploration aur efficiency balance karta hai. Multiple paths consider karta hai bina exhaustive search ke.

Special Tokens and the Mechanism

Problem: Decoder ko kaise pata chalta hai ki kab rukna hai?

Solution: Vocabulary mein ek special end-of-sequence (EOS) token add karo. Model ko train karo ki jab output complete ho tab <EOS> predict kare.

Training data:
Input:  "Hello"
Target: "<SOS> Hola <EOS>"

Inference mein, hum generate karna tab rok dete hain jab model <EOS> output kare ya hum max length hit karein (safety ke liye).

<SOS> kyun? Decoder ko ek initial input chahiye. <SOS> (start-of-sequence) shuruaat signal karta hai.

The Information Bottleneck Problem

Yeh sahi kyun lagta hai: LSTM ka hidden state information aage carry karne ke liye design kiya gaya hai.

Reality yeh hai: Ek fixed-size vector $\mathbf{c}$ (jaise 512 dimensions) ko ek potentially lambi input (jaise 50 tokens, har ek rich meaning ke saath) ke baare mein sab kuch encode karna hota hai. Lambi sequences ke liye, early information bhool jaati hai—LSTM ki memory limited hai.

Fix: Attention mechanisms (next topic). Ek fixed $\mathbf{c}$ ki jagah, decoder har decoding step par ek alag context vector compute karta hai, saare encoder hidden states ko "dekh ke".

Galti ko steel-man karna: Chhoti sequences (5-10 tokens) ke liye, bottleneck itna severe nahin hota. Original seq2seq paper (Sutskever et al., 2014) ne moderate-length sentences par achhe results diye. Problem lambe documents, paragraphs, ya nuanced context ke liye critical ban jaati hai.

Bidirectional Encoder Improvement

Limitation: Token $t$ par ek unidirectional encoder ne sirf $x_1,\dots ,x_t$ dekha hota hai, future tokens nahin.

Enhancement: Encoder ke roop mein ek bidirectional RNN/LSTM use karo: ${\overrightarrow{h}}_t={\text{LSTM}}_{\text{forward}}(x_t,{\overrightarrow{h}}_{t-1})$ ${\overleftarrow{h}}_t={\text{LSTM}}_{\text{backward}}(x_t,{\overleftarrow{h}}_{t+1})$

Concatenate karo: $h_t^{(enc)}=[{\overrightarrow{h}}_t;{\overleftarrow{h}}_t]$

Context vector ban jaata hai: $\mathbf{c}=[h_{T_x}^{\text{forward}};h_1^{\text{backward}}]$

Yeh kyun help karta hai: Encoder ke paas ab har position par poori sequence ki information hoti hai. "Bank" word ko alag samjha ja sakta hai agar uske baad "river" aaye vs. "money".

Input: "The cat sat on the mat. It was very comfortable."
Target: "Cat on mat."

Encoder

Input tokenize karo: ["The", "cat", "sat", "on", "the", "mat", ".", "It", "was", "very", "comfortable", "."]
Bidirectional LSTM se process karo → $\mathbf{c}\in {\mathbb{R}}^{512}$

Decoder

t=1: Input <SOS>, output "Cat" (prob 0.85)
t=2: Input "Cat", output "on" (prob 0.78)
t=3: Input "on", output "mat" (prob 0.82)
t=4: Input "mat", output "." (prob 0.91)
t=5: Input ".", output <EOS> (prob 0.95)

Loss: $-\log (0.85)-\log (0.78)-\log (0.82)-\log (0.91)-\log (0.95)\approx 0.77$

Har step kyun?

• t=1: Decoder seekhta hai sabse important subject pick karna.
• t=2-3: "sat on the" → "on" compress karna seekhta hai.
• t=4: Punctuation well-formed output ka hissa hai.
• t=5: Termination signal.

When to Use Seq2Seq

Ideal nahin hai: Jab input/output alignment known ho (jaise POS tagging—yahaan har input word ko ek tag milta hai; simpler models kaafi hain).

Toh tum ise do steps mein karte ho:

1. Dhyaan se suno poora English sentence aur apne dimag mein idea yaad rakhho (yahi encoder hai jo context vector bana raha hai).
2. Spanish mein bolo, ek word at a time, jo tumne yaad rakha tha use use karte hue (yahi decoder hai).

"Context vector" woh mental note hai jo tumne banaya: "Woh ek khush billi ki baat kar rahe hain jo mat par hai." Phir tum Spanish words bolte ho jo us mental note se match karein, exactly English words se nahin. Isliye lengths alag ho sakti hain—tum ideas translate kar rahe ho, sirf words nahin!

Ya simply: "Samjho, Yaad Rakho, Bolo" (Encoder samajhta hai, Context yaad rakhta hai, Decoder bolta hai).

Connections

• 3.5.05-LSTM-networks – Encoder/decoder typically use LSTMs to handle long sequences
• 3.5.06-GRU-networks – Alternative to LSTM in seq2seq, faster training
• 3.5.08-Attention-mechanism – Solves the context bottleneck by letting decoder "attend" to all encoder states
• 3.5.09-Transformers – Replace RNNs with self-attention, modern seq2seq backbone
• 3.4.03-Backpropagation-through-time – How gradients flow through seq2seq RNNs
• 4.2.01-Beam-search – Decoding strategy for seq2seq
• 2.3.02-Cross-entropy-loss – Loss function for training seq2seq models
• 3.5.01-Recurrent-neural-networks – Foundation of encoder/decoder architecture

#flashcards/ai-ml

Seq2seq model ke do main components kya hain? :: Encoder (input ko context vector mein process karta hai) aur Decoder (context se output generate karta hai)

Concept Map