Encoder-decoder architecture

What is the encoder-decoder architecture?

How does it work? Step-by-step derivation

Step 1: Encoding phase

WHAT: Transform the input sequence into a context representation.

The encoder processes input tokens sequentially:

$h_t^{enc} = f_{enc}(x_t, h_{t-1}^{enc})$

where:

$x_t$ is the input at time $t$
$h_t^{enc}$ is the encoder hidden state
$f_{enc}$ is the recurrent function (LSTM/GRU cell)

WHY this recursion? Each hidden state accumulates information from all previous tokens. By $t=T_x$ , the final hidden state has "seen" the entire sequence.

The context vector is typically:

$\mathbf{c} = h_{T_x}^{enc}$

or sometimes a function of all hidden states:

$\mathbf{c} = q(\{h_1^{enc}, h_2^{enc}, \ldots, h_{T_x}^{enc}\})$

WHY use the last hidden state? In theory, the final state contains a compressed representation of the entire sequence due to the recurrent nature of RNNs.

Step 2: Decoding phase

WHAT: Generate output tokens one at a time, conditioned on the context.

The decoder initializes with the context:

$h_0^{dec} = \mathbf{c}$

Then generates tokens autoregressively:

$h_t^{dec} = f_{dec}(y_{t-1}, h_{t-1}^{dec})$

$P(y_t | y_{<t}, \mathbf{x}) = \text{softmax}(W_o h_t^{dec} + b_o)$

WHY autoregressive? At time $t$ , we need to know what we've already generated ( $y_{<t}$ ) to produce the next token. The decoder's hidden state carries this history.

WHY softmax? It converts raw scores into a probability distribution over the vocabulary: $\sum_{v \in V} P(y_t = v) = 1$ .

Step 3: Training with teacher forcing

Derivation of the loss function:

We want to maximize the likelihood of the correct output sequence:

$\mathcal{L} = \sum_{t=1}^{T_y} \log P(y_t^* | y_{<t}^*, \mathbf{x})$

where $y_t^*$ is the ground-truth token at position $t$ .

WHY this form? We decompose the joint probability using the chain rule:

$P(\mathbf{y}^* | \mathbf{x}) = \prod_{t=1}^{T_y} P(y_t^* | y_{<t}^*, \mathbf{x})$

Taking the log (for numerical stability and converting product to sum):

$\log P(\mathbf{y}^* | \mathbf{x}) = \sum_{t=1}^{T_y} \log P(y_t^* | y_{<t}^*, \mathbf{x})$

Teacher forcing: During training, we feed the ground-truth previous token $y_{t-1}^*$ as input to the decoder at each step, even if the model predicted something else at $t-1$ .

WHY? Faster convergence. Without it, early mistakes compound, and the model sees mostly its own errors during training.

Worked examples

Worked example Example 1: Machine translation (English → French)

Input: "I love AI" → Output: "J'aime l'IA"

Step 1: Encoding

$x_1 = \text{emb}(\text{"I"})$ , $h_1^{enc} = f_{enc}(x_1, h_0^{enc})$
$x_2 = \text{emb}(\text{"love"})$ , $h_2^{enc} = f_{enc}(x_2, h_1^{enc})$
$x_3 = \text{emb}(\text{"AI"})$ , $h_3^{enc} = f_{enc}(x_3, h_2^{enc})$
Context: $\mathbf{c} = h_3^{enc}$ (a512-dim vector, say)

WHY step through each token? The encoder must process the sequence order because "I love AI" has different meaning than "AI love I".

Step 2: Decoding

Initialize: $h_0^{dec} = \mathbf{c}$ , $y_0 = \langle \text{START} \rangle$
$h_1^{dec} = f_{dec}(y_0, h_0^{dec})$ , sample $y_1 = \text{"J'aime"}$ from $P(y_1 | \mathbf{x})$
$h_2^{dec} = f_{dec}(y_1, h_1^{dec})$ , sample $y_2 = \text{"l'IA"}$
$h_3^{dec} = f_{dec}(y_2, h_2^{dec})$ , sample $y_3 = \langle \text{END} \rangle$

WHY feed previous output as next input? The decoder needs context about what it has already said to maintain coherence.

Worked example Example 3: Chatbot response generation

Input: "What is your name?" Output: "I am an AI assistant."

Key insight: The decoder must learn to generate contextually appropriate responses, not just translate.

Math check at $t=2$ (generating "am"): $h_2^{dec} = \tanh(W_{h} h_1^{dec} + W_{hy} \text{emb}(\text{"I"}) + b_h)$

Logits: $z = W_o h_2^{dec} + b_o$ (size = vocabulary size, say50,000)

$P(\text{"am"} | \text{"I"}, \mathbf{x}) = \frac{\exp(z_{\text{am}})}{\sum_{v=1}^{50000} \exp(z_v)}$

WHY this probability? Softmax ensures valid probability distribution and emphasizes the most likely next word through exponentiation.

Common pitfalls and misconceptions

Common mistake Mistake 1: "The context vector contains all information about the input"

Why this feels right: We say the encoder "compresses" the input into $\mathbf{c}$ .

The problem: For long sequences (50+ tokens), a fixed-size vector (512 or 1024 dims) becomes an information bottleneck. Important details get lost, especially from early in the sequence.

Evidence: Seq2seq performance degrades sharply for sentences >20 words.

The fix: Attention mechanism (next topic!) lets the decoder access all encoder hidden states, not just the final one.

Math intuition: Information theory says you need $\log_2(V^{T_x})$ bits to perfectly represent a sequence of length $T_x$ from vocabulary $V$ . For $V=10^4$ , $T_x=50$ : need ~664,000 bits. A 512-dim float32 vector has only16,384 bits. Massive compression = information loss.

Common mistake Mistake 2: "Teacher forcing is always better than free running"

Why this feels right: Teacher forcing makes training faster and more stable.

The problem: Exposure bias. At test time, the decoder never saw its own mistakes during training, so errors compound. If it predicts the wrong token at $t=3$ , it's now in a state it never trained on.

Example: Training with teacher forcing: "I → love → AI" (always correct previous token). Test: Model outputs "I → adore → ???" — it never learned to recover from "adore" instead of "love". The fix:

Scheduled sampling: Gradually mix in model's own predictions during training
Professor forcing: Explicitly train the model to match distributions in both free-running and teacher-forced modes

WHY does this help? The model learns to be robust to its own errors.

Limitations and when it breaks

Long sequences: Context vector bottleneck (solved by attention)
Rare words: Model can't generate unseen vocabulary (solved by subword tokenization, copy mechanism)
Exposure bias: Training-test mismatch (partially addressed by scheduled sampling)
Slow inference: Must decode sequentially, no parallelization (Transformer's decoder can parallelize during training)

Applications

Machine translation (original use case: Sutskever et al., 2014)
Text summarization
Conversational AI / chatbots
Image captioning (CNN encoder + RNN decoder)
Speech recognition (audio encoder + text decoder)
Code generation (natural language → code)

Connections

Prerequisite: 3.5.01-Recurrent-neural-networks, 3.5.03-LSTM-long-short-term-memory, 3.5.04-GRU-gated-recurrent-unit
Enables: 3.5.09-Attention-mechanism, 3.5.10-Transformer-architecture
Related: 3.5.07-Sequence-to-sequence-tasks, 3.3.05-Autoencoders (different: deterministic compression, same-length output)
Contrast: 3.5.02-Backpropagation-through-time (training algorithm), 3.4.06-Beam-search (decoding algorithm)

Recall Feynman explanation (explain to a 12-year-old)

Imagine you're playing the "telephone game" with a robot friend, but the robot can only remember one sentence at a time.

The Encoder is like the robot listening to your entire message: "I want to go to the park play basketball." The robot can't remember every single word, so it writes down a short note: "Kid wants park basketball." That's the context — a super-short summary.

The Decoder is the same robot now telling your message to someone else. It reads its note ("Kid wants park basketball"), then tries to say the message word-by-word in a new language, like Spanish: "Quiero... ir... al... parque..." Each word it says helps it figure out the next word, like building with LEGO blocks one piece at a time. The tricky part? The note might be too short! If your message was really long ("I want to go to the park near my house after lunch to play basketball with my friends and maybe get ice cream after"), the robot's tiny note can't capture everything. It might forget about the ice cream part! That's why scientists invented "attention" (the next topic) — so the robot can look back at your original words while it's speaking, not just its short note.

#flashcards/ai-ml

What are the two main components of an encoder-decoder architecture? :: The encoder (processes input sequence into a fixed-size context vector) and the decoder (generates output sequence conditioned on the context).

What is the context vector in encoder-decoder architecture?

A fixed-size vector representation of the entire input sequence, typically the final hidden state of the encoder:

\mathbf{c} = h_{T_x}^{enc}

Why is the decoding process called "autoregressive"?

Because the decoder generates each output token conditioned on all previously generated tokens:

P(y_t | y_{<t}, \mathbf{x})

What is teacher forcing in seq2seq training?

Feeding the ground-truth previous token as input to the decoder at each step during training, rather than the model's own prediction, to speed up convergence.

What is the main limitation of basic encoder-decoder architecture?

The information bottleneck: a fixed-size context vector cannot adequately represent long input sequences, causing performance degradation.

What is exposure bias in encoder-decoder models?

The mismatch between training (where the model sees ground-truth previous tokens via teacher forcing) and inference (where it sees its own predictions), causing errors to compound at test time.

Why is gredy decoding suboptimal for sequence generation?

Because locally optimal choices (highest probability at each step) don't guarantee globally optimal sequences; a lower-probability token now might enable higher-probability continuations later.

What is beam search and why use it?

A decoding algorithm that maintains

k

candidate sequences (beams) at each step instead of just one, exploring multiple hypotheses to find higher-probability complete sequences than greedy decoding.

Write the loss function for encoder-decoder training :: $\mathcal{L}(\theta) = -\frac{1}{N} \sum_{i=1}^{N} \sum_{t=1}^{T_y^{(i)}} \log P_\theta(y_t^{(i)} | y_{<t}^{(i)}, \mathbf{x}^{(i)})$ (negative log-likelihood).

How does the decoder hidden state update at each step?

h_t^{dec} = f_{dec}(y_{t-1}, h_{t-1}^{dec})

where

y_{t-1}

is the previous output token and

h_{t-1}^{dec}

is the previous hidden state.

What initializes the decoder in encoder-decoder architecture?

The context vector from the encoder:

h_0^{dec} = \mathbf{c} = h_{T_x}^{enc}

Why use softmax for output token probabilities?

To convert raw logits into a valid probability distribution: ensures all probabilities are positive and sum to 1, and emphasizes the most likely tokens through exponentiation.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Encoder-decoder architecture ek powerful technique hai jo variable-length input ko variable-length output mein convert karta hai. Socho tumhe ek English sentence ko Hindi mein translate karna hai - pehle tum pori English sentence ko READ karoge aur samajhoge (yeh ENCODER kaam hai), phir uska meaning yad rakhte hue Hindi mein word-by-word output doge (yeh DECODER ka kaam hai).

Encoder ek RNN/LSTM hota hai jo input sequence ke har token ko process karta hai aur final mein ek CONTEXT VECTOR banata hai - yeh basically pori input ki compressed summary hai, jaise WhatsApp pe long message ko "seen" karke uska gist yad rakhna. Decoder phir iss context vector se start karke, apne previous outputs ko dekhte hue next token predict karta hai, bilkul jaise tum kisi story ko retell karte ho step-by-step.

Iska sabse bada use machine translation mein hai (Google Translate type systems), lekin chatbots, text summarization, aur image captioning mein bhi kaam ata hai. Main challenge yeh hai ki agar input bahut lamba ho (50+ words), toh ek fixed-size context vector sab kuch capture nahi kar pata - information loss ho jata hai. Isi problem ko solve karne ke liye ATTENTION mechanism develophua, jo next topic hai. Training mein hum "teacher forcing" use karte hain jisme model ko correct previous word dete hain (not its own prediction), taki learning fast ho.

What is the encoder-decoder architecture?

How does it work? Step-by-step derivation

Step 1: Encoding phase

Step 2: Decoding phase

Step 3: Training with teacher forcing

Worked examples

Common pitfalls and misconceptions

Limitations and when it breaks

Applications

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Sequence Models

Connections