Encoder vs decoder vs encoder-decoder

4.1.10 · AI-ML › Transformer Architecture

Overview

Teen fundamental transformer architecture patterns alag-alag sequence-to-sequence problems solve karte hain — self-attention mechanisms ko alag configurations mein use karke. Yeh samajhna ki kab kaun sa use karna hai, aapke task ke liye sahi model chunne ke liye bahut zaroori hai.

Sabse badi difference attention masking aur information flow patterns mein hai.

1. Encoder-Only Architecture

Yeh Kaam Kaise Karta Hai (First Principles Se)

Step 1: Bidirectional Context Building

Input sequence $\mathbf{x}=[x_1,x_2,..,x_n]$ ke liye:

$\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V$

Bidirectional KYO? Attention mask saari 1s hai: ${\text{Mask}}_{ij}=1\forall i,j$

Matlab token $i$, token $j$ ko dekh sakta hai chahe position koi bhi ho. Token3 tokens 1, 2, 4, 5, etc. sab dekh sakta hai.

Yeh kyun important hai? "river bank" vs "bank account" mein word "bank" ko alag representations milte hain kyunki yeh SAARE surrounding words ko ek saath dekhta hai.

Step 2: Deep Contextual Representations

$L$ encoder layers ke baad, har token embedding ${\mathbf{h}}_i^{(L)}$ mein POORE sequence ki information hoti hai:

${\mathbf{h}}_i^{(L)}={\text{TransformerBlock}}^{(L)}({\mathbf{h}}_i^{(L-1)},\{{\mathbf{h}}_j^{(L-1)}{\}}_{j=1}^n)$

Multiple layers KYO? Har layer understanding ko refine karti hai:

• Layer 1: "bank" paas mein "river" dekhta hai
• Layer 2: "bank" contextually "water" aur "shore" dekhta hai
• Layer 3: "bank" ki representation ab "river-bank-not-financial-bank" encode karti hai

${\alpha }_{ij}=\frac{\exp (q_i\cdot k_j/\sqrt{d_k})}{{\sum }_{j^{\prime }=1}^n\exp (q_i\cdot k_{j^{\prime }}/\sqrt{d_k})}$

Derivation:

1. Raw similarity se shuru karo: $s_{ij}=q_i\cdot k_j$ (dot product relevance measure karta hai)
2. $\sqrt{d_k}$ se scale karo saturation rokne ke liye: $s_{ij}^{\prime }=s_{ij}/\sqrt{d_k}$ (gradients healthy rakhta hai)
3. SAARI positions par Softmax: ${\alpha }_{ij}=\frac{e^{s_{ij}^{\prime }}}{{\sum }_{j^{\prime }}{e}^{s_{i{j}^{\prime }}^{\prime }}}$ (probability distribution mein normalize karta hai)

Output representation: ${\mathbf{h}}_i={\sum }_{j=1}^n{\alpha }_{ij}{v}_j$

Input tokens: [CLS] This movie was not bad at all ! [SEP]

[CLS] KYO? Special token jiska final representation sentence ka meaning aggregate karta hai.

"not" ke liye Layer 1 attention:

• Attend karta hai: "bad" (0.4), "at" (0.1), "all" (0.2), "was" (0.15), "!" (0.15)
• Kyun? "not" "bad" ko modify karta hai, local context chahiye

"not" ke liye Layer 6 attention:

• Ab [CLS] attend karta hai: "not" (0.3), "bad" (0.25), "!" (0.2)
• Yeh kyun matter karta hai? [CLS] ne seekha ki "not bad" = positive sentiment, bidirectional flow ke through

Final classification: $P(\text{positive})=\text{softmax}(\mathbf{W}{\mathbf{h}}_{\text{[CLS]}}^{(L)})$

Best for: Classification, named entity recognition, question answering (jab answer input mein hi ho)

2. Decoder-Only Architecture

Yeh Kaam Kaise Karta Hai (First Principles Se)

Step 1: Causal Masking

Attention mask ab lower-triangular hai: ${\text{Mask}}_{ij}=\{\begin{array}{ll}1& \text{if }j\le i\\ 0& \text{if }j>i\end{array}$

KYO? Generation ke dauran, position $i-1$ compute karte waqt position $i$ abhi generate nahi hua hota.

Implementation: $\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}+M\right)V$

jahan $M_{ij}=0$ agar $j\le i$ else $-\infty$.

$-\infty$ KYO? Softmax ke baad, $e^{-\infty }=0$, future tokens ko completely block kar deta hai.

Step 2: Autoregressive Generation

Token-by-token generate karo: $P(x_t|x_{<t})=\text{softmax}(\mathbf{W}{\mathbf{h}}_t^{(L)})$

jahan ${\mathbf{h}}_t^{(L)}$ sirf $x_1,...,x_{t-1}$ par depend karta hai.

Full sequence probability: $P(x_1,...,x_n)={\prod }_{t=1}^{n}P(x_t|x_{<t})$

Product KYO? Probability ka chain rule: $P(A,B,C)=P(A)P(B|A)P(C|A,B)$

Generation process:

"which" ke liye Token 3 attention:

• Attend karta hai: "is" (0.1), "Paris" (0.4), "," (0.3), "France" (0.15)
• Future nahi dekh sakta (no look-ahead)

Generation loop:

for t in range(max_length):
    h = decoder(tokens[:t+1])  # Only uses tokens up to t
    logits = output_head(h[t])  # Only last position
    next_token = sample(logits)
    tokens.append(next_token)

Best for: Text generation, code completion, chatbots, koi bhi autoregressive task

3. Encoder-Decoder Architecture

Yeh Kaam Kaise Karta Hai (First Principles Se)

Step 1: Encoder (Bidirectional)

Source sequence $\mathbf{x}=[x_1,...,x_n]$ ko full bidirectional attention se process karta hai: ${\mathbf{h}}_{\text{enc}}^{(L)}=\text{Encoder}(\mathbf{x})$

Har ${\mathbf{h}}_{\text{enc},i}$ saare source tokens dekhta hai.

Step 2: Decoder Self-Attention (Causal)

Target sequence $\mathbf{y}=[y_1,..,y_m]$ ko causal masking se process karta hai: ${\mathbf{z}}_t={\text{SelfAtn}}_{\text{causal}}(y_{\le t})$

Step 3: Cross-Attention (The Key Innovation)

Decoder SAARE encoder outputs ko attend karta hai: ${\mathbf{h}}_{\text{dec},t}=\text{CrossAttn}(Q={\mathbf{z}}_t,K={\mathbf{h}}_{\text{enc}},V={\mathbf{h}}_{\text{enc}})$

Cross-attention ki derivation:

1. Decoder se Query: $Q=W_Q{\mathbf{z}}_t$ (decoder kya "pooch raha hai")
2. Encoder se Keys: $K=W_K{\mathbf{h}}_{\text{enc}}$ (encoder kya "jaanta hai")
3. Encoder se Values: $V=W_V{\mathbf{h}}_{\text{enc}}$ (encoder ki information jo retrieve karni hai)

${\text{CrossAttn}}_t={\sum }_{i=1}^n\frac{\exp (q_t\cdot k_i/\sqrt{d_k})}{{\sum }_{j=1}^n\exp (q_t\cdot k_j/\sqrt{d_k})}{v}_i$

Cross-attention KYO? Decoder position $t$ kisi bhi encoder position ko "dekh" sakta hai, lekin apna future nahi dekh sakta.

Decoder layer (simplified):

1. Causal self-attention: $\mathbf{z}=\text{SelfAttn}(\mathbf{y})$
2. Cross-attention: ${\mathbf{h}}^{\prime }=\text{CrossAttn}(Q=\mathbf{z},K={\mathbf{h}}_{\text{enc}},V={\mathbf{h}}_{\text{enc}})$
3. Feed-forward: ${\mathbf{h}}_{\text{dec}}=\text{FFN}({\mathbf{h}}^{\prime })$

Generation: $P(y_t|y_{<t},\mathbf{x})=\text{softmax}(\mathbf{W}{\mathbf{h}}_{\text{dec},t})$

Encoding phase:

Input: [The, cat, sleeps]
Encoder output: [h_The, h_cat, h_sleps]

Har $h_i$ mein bidirectional context hai.

Decoding step 1: "Le" generate karo

• Decoder input: [START]
• Self-attention: Sirf [START] dekhta hai (left mein kuch nahi)
• Cross-attention: Encoder se query karta hai
  • Attention weights: {The: 0.6, cat: 0.3, sleeps: 0.1}
  • Kyun? "Le" (article) sabse zyada "The" se relate karta hai
• Output: "Le"

Decoding step 2: "chat" generate karo

• Decoder input: [START, Le]
• Self-attention: "chat" "Le" (0.4) aur START (0.6) ko attend karta hai
• Cross-attention:
  • Attention weights: {The: 0.1, cat: 0.8, sleeps: 0.1}
  • Kyun? "chat" "cat" ka translation hai
• Output: "chat"

Decoding step 3: "dort" generate karo

• Decoder input: [START, Le, chat]
• Self-attention: "dort" "Le" (0.2), "chat" (0.5), START (0.3) dekhta hai
• Cross-attention:
  • Attention weights: {The: 0.05, cat: 0.15, sleeps: 0.8}
  • Kyun? "dort" "sleeps" ka translation hai
• Output: "dort"

Encoder-decoder yahan kyun jeet ta hai?

1. Encoder English sentence ki poori understanding build karta hai
2. Decoder French word-by-word generate karta hai
3. Cross-attention source aur target ko align karta hai (attention as soft alignment)

Best for: Translation, summarization, question answering (answers generate karna), koi bhi seq2seq jahan input/output alag ho

Architecture Comparison Table

Fix: Encoders samajhne mein ZYADA powerful hote hain kyunki woh dono directions dekhte hain. Jahan deep comprehension chahiye (semantic search, subtle cues wali classification), encoders jeette hain.

Example: BERT sentiment same size ke GPT-style models se better classify karta hai kyunki "not bad" ke liye encoding ke dauran DONO words simultaneously dekhna padta hai.

Fix: Cross-attention asymmetric hai:

• Query decoder se aata hai (decoder sawaal pooch raha hai)
• Key/Value encoder se aata hai (encoder jawab de raha hai)

Regular attention symmetric hota hai (Q, K, V sab same sequence se). Cross-attention information flow create karta hai: encoder → decoder.

Example: "dort" generate karte waqt, decoder encoder se QUERY karta hai ki "main kaunsa English word translate kar raha hoon?" Encoder ki "sleeps" representation high attention weight ke through RESPOND karti hai.

Fix: Encoder-decoder mein 2× parameters hote hain aur yeh overkill hai jab dono directions ki zaroorat nahi.

Rules:

• Same input/output modality + generation → Decoder-only (GPT)
• Generation ke bina understanding → Encoder-only (BERT)
• Alag input/output ya summarization → Encoder-decoder (T5)

Mathematical Connections

Unified View: Teeno general transformer equation ke special cases hain:

${\mathbf{h}}_i^{(l+1)}=\text{FFN}\left({\sum }_{j\in \mathcal{N}(i)}{\alpha }_{ij}^{(l)}{\mathbf{v}}_j^{(l)}\right)$

jahan $\mathcal{N}(i)$ woh neighborhood hai jise token $i$ attend kar sakta hai:

• Encoder: $\mathcal{N}(i)=\{1,2,...,n\}$ (saare tokens)
• Decoder: $\mathcal{N}(i)=\{1,2,...,i\}$ (causal)
• Encoder-Decoder: $\mathcal{N}(i)=\{1,..,i\}$ (self) $\cup$ $\{1,...,n_{\text{enc}}\}$ (cross)

Helper 1 (Encoder-only): "Super Reader"

• Tumhara sawaal answer karne se pehle POORI kitaab padhta hai
• Shuruaat samajhne se pehle ending bhi dekh leta hai
• Best at: "Yeh kahaani kis baare mein hai?" (understanding tasks)
• Nayi kahaaniyaan nahi likh sakta, sirf existing samajhta hai

Helper 2 (Decoder-only): "Story Writer"

• Kahaaniyaan word by word likhta hai, kabhi aage nahi dekhta
• Sirf jo likh chuka hai wahi yaad rehta hai
• Best at: "Yeh kahaani aage badhao..." (generation tasks)
• Naya content likhta hai lekin gehri reading nahi karta

Helper 3 (Encoder-Decoder): "Translator"

• Pehle POORA French sentence padhta hai (encoder)
• Phir English word-by-word likhta hai (decoder)
• Har English word likhte waqt French ko wapas dekhta hai (cross-attention)
• Best at: "Ise kuch aur mein badlo" (transformation tasks)

Sabse badi difference? Kaam karte waqt yeh kahan dekh sakte hain!

Ya: "Bidirectional Brain, Causal Creator, Cross-attention Converter"

Connections

• Self-Attention Mechanism - Teeno architectures ki foundation
• Positional Encoding - Saare variants sequence order kaise handle karte hain
• Multi-Head Attention - Encoder, decoder, aur cross-attention mein use hota hai
• Masked Language Modeling - Encoder-only ka training objective
• Causal Language Modeling - Decoder-only ka training objective
• Sequenceto-Sequence Models - Encoder-decoder ka problem class
• Attention Masking - Alag masks har architecture define karte hain
• BERT - Canonical encoder-only model
• GPT - Canonical decoder-only model
• T5 - Canonical encoder-decoder model
• Cross-Attention - Encoder-decoder ka unique component
• Autoregressive Generation - Decoders aur encoder-decoders kaise generate karte hain

#flashcards/ai-ml

Encoder-only aur decoder-only architectures mein key attention difference kya hai? :: Encoder-only bidirectional attention use karta hai (har token saare tokens dekhta hai), jabki decoder-only causal attention use karta hai (har token sirf pehle ke tokens dekhta hai). Yeh attention masking se enforce hota hai.

Causal attention mask mathematically derive karo :: Mask_{ij} = 1 agar j ≤ i, else 0. Implement aise: Attention(Q,K,V) = softmax(QK^T/√d_k + M)V, jahan M_{ij} = 0 agar j≤i, else -∞. Softmax ke baad -∞ 0 ban jaata hai, future tokens block ho jaate hain.

Concept Map