Encoder-decoder architecture — Hinglish notes

3.5.8 · AI-ML › Sequence Models

Encoder-decoder architecture kya hai?

Yeh kaam kaise karta hai? Step-by-step derivation

Step 1: Encoding phase

KYA: Input sequence ko ek context representation mein transform karo.

Encoder input tokens ko sequentially process karta hai:

$h_{t}^{e n c} = f_{e n c} (x_{t}, h_{t - 1}^{e n c})$

jahan:

$x_{t}$ time $t$ par input hai
$h_{t}^{e n c}$ encoder hidden state hai
$f_{e n c}$ recurrent function hai (LSTM/GRU cell)

YEH RECURSION KYUN? Har hidden state saare pichle tokens ki information accumulate karta hai. $t = T_{x}$ tak, final hidden state ne poori sequence "dekh" li hoti hai.

Context vector typically hota hai:

$c = h_{T_{x}}^{e n c}$

ya kabhi kabhi saare hidden states ka ek function:

$c = q ({h_{1}^{e n c}, h_{2}^{e n c}, \dots, h_{T_{x}}^{e n c}})$

LAST HIDDEN STATE KYUN USE KARO? Theory mein, RNNs ki recurrent nature ki wajah se final state poori sequence ki ek compressed representation contain karta hai.

Step 2: Decoding phase

KYA: Output tokens ek-ek karke generate karo, context par conditioned hokar.

Decoder context se initialize hota hai:

$h_{0}^{d ec} = c$

Phir tokens autoregressively generate karta hai:

$h_{t}^{d ec} = f_{d ec} (y_{t - 1}, h_{t - 1}^{d ec})$

$P (y_{t} ∣ y_{< t}, x) = softmax (W_{o} h_{t}^{d ec} + b_{o})$

AUTOREGRESSIVE KYUN? Time $t$ par, humein pata hona chahiye ki humne pehle kya generate kiya hai ( $y_{< t}$ ) taaki agla token produce kar sakein. Decoder ka hidden state yeh history carry karta hai.

SOFTMAX KYUN? Yeh raw scores ko vocabulary par ek probability distribution mein convert karta hai: $\sum_{v \in V} P (y_{t} = v) = 1$ .

Step 3: Training with teacher forcing

Loss function ki derivation:

Hum correct output sequence ki likelihood maximize karna chahte hain:

$L = \sum_{t = 1}^{T_{y}} lo g P (y_{t}^{*} ∣ y_{< t}^{*}, x)$

jahan $y_{t}^{*}$ position $t$ par ground-truth token hai.

YEH FORM KYUN? Hum joint probability ko chain rule use karke decompose karte hain:

$P (y^{*} ∣ x) = \prod_{t = 1}^{T_{y}} P (y_{t}^{*} ∣ y_{< t}^{*}, x)$

Log lete hain (numerical stability ke liye aur product ko sum mein convert karne ke liye):

$lo g P (y^{*} ∣ x) = \sum_{t = 1}^{T_{y}} lo g P (y_{t}^{*} ∣ y_{< t}^{*}, x)$

Teacher forcing: Training ke dauran, hum har step par decoder ko ground-truth previous token $y_{t - 1}^{*}$ as input dete hain, chahe model ne $t - 1$ par kuch aur hi predict kiya ho.

KYUN? Faster convergence. Iske bina, early mistakes compound hoti hain, aur model training mein mostly apni khud ki galtiyan dekhta hai.

Worked examples

Worked example Example 1: Machine translation (English → French)

Input: "I love AI" → Output: "J'aime l'IA"

Step 1: Encoding

$x_{1} = emb ("I")$ , $h_{1}^{e n c} = f_{e n c} (x_{1}, h_{0}^{e n c})$
$x_{2} = emb ("love")$ , $h_{2}^{e n c} = f_{e n c} (x_{2}, h_{1}^{e n c})$
$x_{3} = emb ("AI")$ , $h_{3}^{e n c} = f_{e n c} (x_{3}, h_{2}^{e n c})$
Context: $c = h_{3}^{e n c}$ (maano ek 512-dim vector)

HAR TOKEN STEP-THROUGH KYUN? Encoder ko sequence order process karna padta hai kyunki "I love AI" ka meaning "AI love I" se alag hai.

Step 2: Decoding

Initialize: $h_{0}^{d ec} = c$ , $y_{0} = ⟨ START ⟩$
$h_{1}^{d ec} = f_{d ec} (y_{0}, h_{0}^{d ec})$ , sample $y_{1} = "J’aime"$ from $P (y_{1} ∣ x)$
$h_{2}^{d ec} = f_{d ec} (y_{1}, h_{1}^{d ec})$ , sample $y_{2} = "l’IA"$
$h_{3}^{d ec} = f_{d ec} (y_{2}, h_{2}^{d ec})$ , sample $y_{3} = ⟨ END ⟩$

PREVIOUS OUTPUT KO NEXT INPUT KYUN FEED KARO? Decoder ko context chahiye ki usne pehle kya kaha hai taaki coherence maintain ho sake.

Common pitfalls aur misconceptions

Common mistake Mistake 1: "Context vector input ke baare mein saari information contain karta hai"

Yeh sahi kyun lagta hai: Hum kehte hain encoder input ko $c$ mein "compress" karta hai.

Problem: Long sequences (50+ tokens) ke liye, ek fixed-size vector (512 ya 1024 dims) ek information bottleneck ban jaata hai. Important details kho jaati hain, khaaskar sequence ke shuru ki.

Evidence: Seq2seq performance 20 words se zyada sentences ke liye sharply degrade hoti hai.

Fix: Attention mechanism (agla topic!) decoder ko sirf final state nahi, balki saare encoder hidden states access karne deta hai.

Math intuition: Information theory kehti hai ek length $T_{x}$ ki sequence perfectly represent karne ke liye vocabulary $V$ se $lo g_{2} (V^{T_{x}})$ bits chahiye. $V = 1 0^{4}$ , $T_{x} = 50$ ke liye: ~664,000 bits chahiye. Ek 512-dim float32 vector mein sirf 16,384 bits hain. Massive compression = information loss.

Common mistake Mistake 2: "Teacher forcing hamesha free running se better hai"

Yeh sahi kyun lagta hai: Teacher forcing training ko faster aur more stable banata hai.

Problem: Exposure bias. Test time par, decoder ne training mein apni galtiyan kabhi nahi dekhi, isliye errors compound hoti hain. Agar $t = 3$ par wrong token predict karta hai, toh ab woh ek aisi state mein hai jis par usne kabhi train nahi kiya.

Example: Teacher forcing se training: "I → love → AI" (hamesha correct previous token). Test: Model output karta hai "I → adore → ???" — usne kabhi nahi seekha ki "love" ki jagah "adore" aane par kaise recover karein. Fix:

Scheduled sampling: Training ke dauran gradually model ki apni predictions mix in karo
Professor forcing: Model ko explicitly train karo ki free-running aur teacher-forced dono modes mein distributions match kare

YEH HELP KYUN KARTA HAI? Model apni khud ki galtiyon ke liye robust hona seekhta hai.

Limitations aur kab yeh fail karta hai

Long sequences: Context vector bottleneck (attention se solve hota hai)
Rare words: Model unseen vocabulary generate nahi kar sakta (subword tokenization, copy mechanism se solve hota hai)
Exposure bias: Training-test mismatch (scheduled sampling se partially address hota hai)
Slow inference: Sequentially decode karna padta hai, koi parallelization nahi (Transformer ka decoder training ke dauran parallelize kar sakta hai)

Applications

Machine translation (original use case: Sutskever et al., 2014)
Text summarization
Conversational AI / chatbots
Image captioning (CNN encoder + RNN decoder)
Speech recognition (audio encoder + text decoder)
Code generation (natural language → code)

Connections

Prerequisite: 3.5.01-Recurrent-neural-networks, 3.5.03-LSTM-long-short-term-memory, 3.5.04-GRU-gated-recurrent-unit
Enables: 3.5.09-Attention-mechanism, 3.5.10-Transformer-architecture
Related: 3.5.07-Sequence-to-sequence-tasks, 3.3.05-Autoencoders (alag: deterministic compression, same-length output)
Contrast: 3.5.02-Backpropagation-through-time (training algorithm), 3.4.06-Beam-search (decoding algorithm)

Recall Feynman explanation (ek 12-saal ke bacche ko explain karo)

Socho tum ek robot dost ke saath "telephone game" khel rahe ho, lekin robot ek baar mein sirf ek sentence yaad rakh sakta hai.

Encoder jaisa robot teri poori baat sun raha hota hai: "I want to go to the park play basketball." Robot har ek word yaad nahi rakh sakta, toh ek short note likhta hai: "Kid wants park basketball." Yahi context hai — ek super-short summary.

Decoder wahi robot hai jo ab teri baat kisi aur ko sunaa raha hai. Woh apna note padhta hai ("Kid wants park basketball"), phir kisi nayi language mein, jaise Spanish mein, word-by-word message bolne ki koshish karta hai: "Quiero... ir... al... parque..." Har word jo woh bolta hai, usse agla word figure out karne mein help karta hai, jaise LEGO blocks ek-ek karke lagana. Tricky part kya hai? Note bahut chhota ho sakta hai! Agar teri message bahut lambi thi ("I want to go to the park near my house after lunch to play basketball with my friends and maybe get ice cream after"), toh robot ka tiny note sab capture nahi kar sakta. Woh ice cream waala part bhool sakta hai! Isliye scientists ne "attention" invent kiya (agla topic) — taaki robot teri original baaton ko dobara dekh sake jab woh bol raha ho, sirf apne chhote note par nahi.

#flashcards/ai-ml

Encoder-decoder architecture ke do main components kya hain? :: Encoder (input sequence ko ek fixed-size context vector mein process karta hai) aur decoder (context par conditioned hokar output sequence generate karta hai).

Encoder-decoder architecture mein context vector kya hota hai?

Poori input sequence ki ek fixed-size vector representation, typically encoder ka final hidden state:

c = h_{T_{x}}^{e n c}

Decoding process ko "autoregressive" kyun kaha jaata hai?

Kyunki decoder har output token saare pehle generate kiye tokens par conditioned hokar generate karta hai:

P (y_{t} ∣ y_{< t}, x)

Seq2seq training mein teacher forcing kya hota hai?

Training ke dauran har step par decoder ko model ki apni prediction ki jagah ground-truth previous token as input dena, convergence speed up karne ke liye.

Basic encoder-decoder architecture ki main limitation kya hai?

Information bottleneck: ek fixed-size context vector lambi input sequences ko adequately represent nahi kar sakta, jisse performance degrade hoti hai.

Encoder-decoder models mein exposure bias kya hota hai?

Training (jahan model teacher forcing se ground-truth previous tokens dekhta hai) aur inference (jahan woh apni predictions dekhta hai) ke beech ka mismatch, jisse test time par errors compound hoti hain.

Greedy decoding sequence generation ke liye suboptimal kyun hota hai?

Kyunki locally optimal choices (har step par highest probability) globally optimal sequences guarantee nahi karte; ek lower-probability token abhi baad mein higher-probability continuations enable kar sakta hai.

Beam search kya hai aur ise kyun use karte hain?

Ek decoding algorithm jo har step par sirf ek ki jagah

k

candidate sequences (beams) maintain karta hai, greedy decoding se zyada higher-probability complete sequences dhundhne ke liye multiple hypotheses explore karta hai.

Encoder-decoder training ke liye loss function likho :: $L (θ) = - \frac{1}{N} \sum_{i = 1}^{N} \sum_{t = 1}^{T_{y}^{(i)}} lo g P_{θ} (y_{t}^{(i)} ∣ y_{< t}^{(i)}, x^{(i)})$ (negative log-likelihood).

Decoder ka hidden state har step par kaise update hota hai?

h_{t}^{d ec} = f_{d ec} (y_{t - 1}, h_{t - 1}^{d ec})

jahan

y_{t - 1}

previous output token hai aur

h_{t - 1}^{d ec}

previous hidden state hai.

Encoder-decoder architecture mein decoder ko kya initialize karta hai?

Encoder ka context vector:

h_{0}^{d ec} = c = h_{T_{x}}^{e n c}

Output token probabilities ke liye softmax kyun use karte hain?

Raw logits ko ek valid probability distribution mein convert karne ke liye: ensure karta hai ki saari probabilities positive hain aur sum to 1 hain, aur exponentiation ke through most likely tokens ko emphasize karta hai.