Encoder vs decoder vs encoder-decoder

Overview

The three fundamental transformer architecture patterns solve different sequence-to-sequence problems by using self-attention mechanisms in different configurations. Understanding when to use each is crucial for choosing the right model for your task.

The key difference is in attention masking and information flow patterns.

1. Encoder-Only Architecture

How It Works (From First Principles)

Step 1: Bidirectional Context Building

For input sequence $\mathbf{x} = [x_1, x_2, .., x_n]$:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$

WHY bidirectional? The attention mask is all1s: $\text{Mask}_{ij} = 1 \quad \forall i,j$

This means token $i$ can see token $j$ regardless of position. Token3 sees tokens 1, 2, 4, 5, etc.

WHY does this matter? The word "bank" in "river bank" vs "bank account" gets different representations because it sees ALL surrounding words simultaneously.

Step 2: Deep Contextual Representations

After $L$ encoder layers, each token embedding $\mathbf{h}_i^{(L)}$ contains information from the ENTIRE sequence:

$\mathbf{h}_i^{(L)} = \text{TransformerBlock}^{(L)}(\mathbf{h}_i^{(L-1)}, \{\mathbf{h}_j^{(L-1)}\}_{j=1}^n)$

WHY multiple layers? Each layer refines understanding:

• Layer 1: "bank" sees "river" nearby
• Layer 2: "bank" sees "water" and "shore" contextually
• Layer 3: "bank" representation now encodes "river-bank-not-financial-bank"

$\alpha_{ij} = \frac{\exp(q_i \cdot k_j / \sqrt{d_k})}{\sum_{j'=1}^{n} \exp(q_i \cdot k_{j'} / \sqrt{d_k})}$

Derivation:

1. Start with raw similarity: $s_{ij} = q_i \cdot k_j$ (dot product measures relevance)
2. Scale by $\sqrt{d_k}$ to prevent saturation: $s_{ij}' = s_{ij} / \sqrt{d_k}$ (keeps gradients healthy)
3. Softmax over ALL positions: $\alpha_{ij} = \frac{e^{s_{ij}'}}{\sum_{j'} e^{s_{ij'}'}}$ (normalizes to probability distribution)

Output representation: $\mathbf{h}_i = \sum_{j=1}^{n} \alpha_{ij} v_j$

Input tokens: [CLS] This movie was not bad at all ! [SEP]

WHY [CLS]? Special token whose final representation aggregates sentence meaning.

Layer 1 attention for "not":

• Attends to: "bad" (0.4), "at" (0.1), "all" (0.2 "was" (0.15), "!" (0.15)
• Why? "not" modifies "bad", needs local context

Layer 6 attention for "not":

• Now [CLS] attends to: "not" (0.3), "bad" (0.25), "!" (0.2)
• Why this matters? [CLS] learned "not bad" = positive sentiment through bidirectional flow

Final classification: $P(\text{positive}) = \text{softmax}(\mathbf{W} \mathbf{h}_{\text{[CLS]}}^{(L)})$

Best for: Classification, named entity recognition, question answering (when answer is IN the input)

2. Decoder-Only Architecture

How It Works (From First Principles)

Step 1: Causal Masking

The attention mask is now lower-triangular: $\text{Mask}_{ij} = \begin{cases} 1 & \text{if } j \leq i \\ 0 & \text{if } j > i \end{cases}$

WHY? During generation, position $i$ hasn't been generated yet when computing position $i-1$.

Implementation: $\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}} + M\right)V$

where $M_{ij} = 0$ if $j \leq i$ else $-\infty$.

WHY $-\infty$? After softmax, $e^{-\infty} = 0$, completely blocking future tokens.

Step 2: Autoregressive Generation

Generate token-by-token: $P(x_t | x_{<t}) = \text{softmax}(\mathbf{W} \mathbf{h}_t^{(L)})$

where $\mathbf{h}_t^{(L)}$ only depends on $x_1, ..., x_{t-1}$.

Full sequence probability: $P(x_1, ..., x_n) = \prod_{t=1}^{n} P(x_t | x_{<t})$

WHY product? Chain rule of probability: $P(A,B,C) = P(A)P(B|A)P(C|A,B)$

Generation process:

| Step | Input | Attention | Output Token | Why? | |------|-------|-----------|--------------| | 1 | [The capital of France is] | Each token sees only left | "Paris" | Pattern: "[capital of X is Y]" learned from training | | 2 | [The capital of France is Paris] | "Paris" only sees prompt | "," | Punctuation pattern after city names | | 3 | [.., Paris,] | "," sees all previous | "which" | Common continuation "Paris, which..." |

Token 3 attention for "which":

• Attends to: "is" (0.1), "Paris" (0.4), "," (0.3), "France" (0.15)
• Cannot see future (no look-ahead)

Generation loop:

for t in range(max_length):
    h = decoder(tokens[:t+1])  # Only uses tokens up to t
    logits = output_head(h[t])  # Only last position
    next_token = sample(logits)
    tokens.append(next_token)

Best for: Text generation, code completion, chatbots, any autoregressive task

3. Encoder-Decoder Architecture

How It Works (From First Principles)

Step 1: Encoder (Bidirectional)

Processes source sequence $\mathbf{x} = [x_1, ..., x_n]$ with full bidirectional attention: $\mathbf{h}_{\text{enc}}^{(L)} = \text{Encoder}(\mathbf{x})$

Each $\mathbf{h}_{\text{enc}, i}$ sees all source tokens.

Step 2: Decoder Self-Attention (Causal)

Processes target sequence $\mathbf{y} = [y_1, .., y_m]$ with causal masking: $\mathbf{z}_t = \text{SelfAtn}_{\text{causal}}(y_{\leq t})$

Step 3: Cross-Attention (The Key Innovation)

Decoder attends to ALL encoder outputs: $\mathbf{h}_{\text{dec}, t} = \text{CrossAttn}(Q=\mathbf{z}_t, K=\mathbf{h}_{\text{enc}}, V=\mathbf{h}_{\text{enc}})$

Derivation of cross-attention:

1. Query from decoder: $Q = W_Q \mathbf{z}_t$ (what decoder is "asking about")
2. Keys from encoder: $K = W_K \mathbf{h}_{\text{enc}}$ (what encoder "knows")
3. Values from encoder: $V = W_V \mathbf{h}_{\text{enc}}$ (encoder's information to retrieve)

$\text{CrossAttn}_t = \sum_{i=1}^{n} \frac{\exp(q_t \cdot k_i / \sqrt{d_k})}{\sum_{j=1}^{n} \exp(q_t \cdot k_j / \sqrt{d_k})} v_i$

WHY cross-attention? Decoder position $t$ can "look at" any encoder position, but still can't see its own future.

Decoder layer (simplified):

1. Causal self-attention: $\mathbf{z} = \text{SelfAttn}(\mathbf{y})$
2. Cross-attention: $\mathbf{h}' = \text{CrossAttn}(Q=\mathbf{z}, K=\mathbf{h}_{\text{enc}}, V=\mathbf{h}_{\text{enc}})$
3. Feed-forward: $\mathbf{h}_{\text{dec}} = \text{FFN}(\mathbf{h}')$

Generation: $P(y_t | y_{<t}, \mathbf{x}) = \text{softmax}(\mathbf{W} \mathbf{h}_{\text{dec}, t})$

Encoding phase:

Input: [The, cat, sleeps]
Encoder output: [h_The, h_cat, h_sleps]

Each $h_i$ contains bidirectional context.

Decoding step 1: Generate "Le"

• Decoder input: [START]
• Self-attention: Only sees [START] (nothing to left)
• Cross-attention: Queries encoder
  • Attention weights: {The: 0.6, cat: 0.3, sleeps: 0.1}
  • Why? "Le" (article) relates most to "The"
• Output: "Le"

Decoding step 2: Generate "chat"

• Decoder input: [START, Le]
• Self-attention: "chat" attends to "Le" (0.4) and START (0.6)
• Cross-attention:
  • Attention weights: {The: 0.1, cat: 0.8, sleeps: 0.1}
  • Why? "chat" is translation of "cat"
• Output: "chat"

Decoding step 3: Generate "dort"

• Decoder input: [START, Le, chat]
• Self-attention: "dort" sees "Le" (0.2), "chat" (0.5), START (0.3)
• Cross-attention:
  • Attention weights: {The: 0.05, cat: 0.15, sleeps: 0.8}
  • Why? "dort" translates "sleps"
• Output: "dort"

Why encoder-decoder wins here?

1. Encoder builds full understanding of English sentence
2. Decoder generates French word-by-word
3. Cross-attention aligns source and target (attention as soft alignment)

Best for: Translation, summarization, question answering (generating answers), any seq2seq with different input/output

Architecture Comparison Table

| Aspect | Encoder-Only | Decoder-Only | Encoder-Decoder | |--------|--------------|-----------------| | Attention Type | Bidirectional | Causal (unidirectional) | Both + cross-attention | | Token Visibility | All tokens | Only previous tokens | Encoder: all; Decoder: previous + encoder | | Training Objective | Masked LM (MLM) | Next token prediction | Seq2seq loss | | Generation | ❌ No | ✅ Yes (autoregressive) | ✅ Yes (autoregressive) | | Understanding | ✅ Excellent | Good✅ Excellent (encoder side) | | Best For | Classification, NER, embedding | Text generation, chat | Translation, summarization | | Examples | BERT, RoBERTa | GPT, LaMA, Claude | T5, BART, mT5 |

The fix: Encoders are MORE powerful at understanding because they see both directions. For tasks requiring deep comprehension (semantic search, classification with subtle cues), encoders win.

Example: BERT classifies sentiment better than GPT-style models of same size because "not bad" requires seeing BOTH words simultaneously during encoding.

The fix: Cross-attention is asymetric:

• Query from decoder (decoder asking questions)
• Key/Value from encoder (encoder providing answers)

Regular attention is symmetric (Q, K, V all from same sequence). Cross-attention creates information flow: encoder → decoder.

Example: When generating "dort", the decoder QUERIES the encoder for "which English word am I translating?" The encoder's "sleps" representation RESPONDS through high attention weight.

The fix: Encoder-decoder has 2× parameters and is overkill when you don't need both directions.

Rules:

• Same input/output modality + generation → Decoder-only (GPT)
• Understanding without generation → Encoder-only (BERT)
• Different input/output or summarization → Encoder-decoder (T5)

Mathematical Connections

Unified View: All three are special cases of the general transformer equation:

$\mathbf{h}_{i}^{(l+1)} = \text{FFN}\left(\sum_{j \in \mathcal{N}(i)} \alpha_{ij}^{(l)} \mathbf{v}_j^{(l)}\right)$

where $\mathcal{N}(i)$ is the neighborhood that token $i$ can attend to:

• Encoder: $\mathcal{N}(i) = \{1, 2, ..., n\}$ (all tokens)
• Decoder: $\mathcal{N}(i) = \{1, 2, ..., i\}$ (causal)
• Encoder-Decoder: $\mathcal{N}(i) = \{1, .., i\}$ (self) $\cup$ $\{1, ..., n_{\text{enc}}\}$ (cross)

Helper 1 (Encoder-only): The "Super Reader"

• Reads the WHOLE book before answering your question
• Sees the ending before understanding the beginning
• Best at: "What's this story about?" (understanding tasks)
• Can't write new stories, just understands existing ones

Helper 2 (Decoder-only): The "Story Writer"

• Writes stories word by word, never peeking ahead
• Only remembers what they've already written
• Best at: "Continue this story..." (generation tasks)
• Writes new content but doesn't do deep reading

Helper 3 (Encoder-Decoder): The "Translator"

• First, reads the WHOLE French sentence (encoder)
• Then, writes English word-by-word (decoder)
• While writing each English word, looks back at the French (cross-attention)
• Best at: "Change this into something else" (transformation tasks)

The key difference? Where they can look while working!

Or: "Bidirectional Brain, Causal Creator, Cross-attention Converter"

Connections

• Self-Attention Mechanism - Foundation for all three architectures
• Positional Encoding - How all variants handle sequence order
• Multi-Head Attention - Used in encoder, decoder, and cross-attention
• Masked Language Modeling - Training objective for encoder-only
• Causal Language Modeling - Training objective for decoder-only
• Sequenceto-Sequence Models - Problem class solved by encoder-decoder
• Attention Masking - Different masks define each architecture
• BERT - Canonical encoder-only model
• GPT - Canonical decoder-only model
• T5 - Canonical encoder-decoder model
• Cross-Attention - Unique component of encoder-decoder
• Autoregressive Generation - How decoders and encoder-decoders generate

#flashcards/ai-ml

What is the key attention difference between encoder-only and decoder-only architectures? :: Encoder-only uses bidirectional attention (each token sees all tokens), while decoder-only uses causal attention (each token only sees previous tokens). This is enforced through attention masking.

Derive the causal attention mask mathematically :: Mask_{ij} = 1 if j ≤ i, else 0. Implemented as: Attention(Q,K,V) = softmax(QK^T/√d_k + M)V, where M_{ij} = 0 if j≤i, else -∞. The -∞ becomes 0 after softmax, blocking future tokens.

Concept Map

Hinglish (regional understanding)