Sequence-to-sequence models

What Problem Does Seq2Seq Solve?

The challenge: Standard neural networks expect fixed input/output sizes. But in machine translation, summarization, chatbots, the input "Bonjour" → output "Hello" has 1 token → 1 token, while "Comment ça va?" → "How are you doing?" has 3 → 4 tokens.

The breakthrough: Decouple input processing from output generation.

• Encoder: Reads the entire input sequence $x_1, x_2, \dots, x_{T_x}$ and produces a context vector $\mathbf{c}$ (the "thought").
• Decoder: Generates output sequence $y_1, y_2, \dots, y_{T_y}$ conditioned on $\mathbf{c}$, one token at a time.

Why this works: The context vector is a learned, compressed representation. The decoder learns to "unfold" this representation into the target language/format.

Architecture Derivation from First Principles

Step 1: The Encoder

We need to process a variable-length input. What structure handles sequences? RNNs, LSTMs, or GRUs.

Let the encoder be an LSTM. At each time step $t$: $h_t^{(enc)} = \text{LSTM}(x_t, h_{t-1}^{(enc)})$

After processing all $T_x$ tokens, the final hidden state $h_{T_x}^{(enc)}$ contains information about the entire input. We set: $\mathbf{c} = h_{T_x}^{(enc)}$

Why the final state? In an LSTM, information flows forward. The last hidden state has "seen" all previous tokens (though it might forget early ones—this motivates attention later).

Step 2: The Decoder

The decoder generates the output sequence. At each decoding step $t$: $h_t^{(dec)} = \text{LSTM}(\mathbf{c}, y_{t-1}, h_{t-1}^{(dec)})$

Wait, what's $y_{t-1}$? The previous output token. During training, we use teacher forcing: feed the ground-truth previous token. During inference, we feed the model's own prediction.

The decoder's hidden state $h_t^{(dec)}$ is passed through a softmax layer to predict the next token: $P(y_t \mid y_{<t}, \mathbf{c}) = \text{softmax}(W_o h_t^{(dec)} + b_o)$

Why softmax? We're choosing one token from a vocabulary $V$, so we need a probability distribution over $|V|$ classes.

The full sequence probability (assuming conditional independence given $\mathbf{c}$): $P(y_1, \dots, y_{T_y} \mid x_1, \dots, x_{T_x}) = \prod_{t=1}^{T_y} P(y_t \mid y_{<t}, \mathbf{c})$

Step 3: Training with Cross-Entropy

We have ground-truth pairs $(x, y)$. The loss for one example: $\mathcal{L} = -\sum_{t=1}^{T_y} \log P(y_t^* \mid y_{<t}^*, \mathbf{c})$

where $y_t^*$ is the true target token at position $t$.

Why negative log-likelihood? We want to maximize the probability of the correct sequence. Maximizing $\log P(y \mid x)$ is equivalent to minimizing $-\log P(y \mid x)$.

Backpropagation: Gradients flow backward through the decoder, into $\mathbf{c}$, then through the encoder. This is backpropagation through time (BPTT) across two RNNs.

Training vs. Inference: The Critical Difference

Training (Teacher Forcing)

Input:  "I love AI" 
Target: "<SOS> Me encanta IA <EOS>"

Decoder sees at t=1: <SOS> (from target)
Decoder predicts: "Me" (loss if wrong)

Decoder sees at t=2: "Me" (ground truth, not prediction!)
Decoder predicts: "encanta"
... and so on

Why teacher forcing? Speeds up training. The model gets the correct context at each step, rather than compounding errors.

Inference (Autoregressive Generation)

Input: "I love AI"
Decoder starts with <SOS>

t=1: Decoder predicts "Me" (from <SOS>)
t=2: Decoder uses "Me" to predict "encanta"
t=3: Decoder uses "encanta" to predict "IA"
t=4: Decoder predicts <EOS>, stop.

The exposure bias problem: At training, the decoder never sees its own mistakes. At inference, one wrong prediction can cascade. This is a known weakness of teacher forcing.

Worked Example: English → Spanish Translation

Input: "I am happy"
Target: "Estoy feliz"

Encoding Phase

Vocabulary: {I, am, happy} → {0, 1, 2}
Embed: $x_1 = E[0], x_2 = E[1], x_3 = E[2]$

Encoder LSTM (simplified, $d_h = 4$):

h_0 = [0,0,0]
h_1 = LSTM(x_1, h_0) → [0.2, -0.1, 0.5, 0.3]
h_2 = LSTM(x_2, h_1) → [0.4, 0.2, 0.1, -0.2]
h_3 = LSTM(x_3, h_2) → [0.1, 0.6, -0.3, 0.4]

Context vector c = h_3 = [0.1, 0.6, -0.3, 0.4]

Decoding Phase

Target vocab: {, Estoy, feliz, } → {0, 1, 2, 3}

Initialize decoder with $\mathbf{c}$:

Input: y_0 = <SOS> (token 0)
h_1^(dec) = LSTM(c, E[0], h_0^(dec))
Output: P(y_1|c) = softmax(W_o h_1^(dec))
       = [0.01, 0.92, 0.05, 0.02]  # Predicts "Estoy" (token 1)

Input: y_1 = "Estoy" (token 1, teacher forcing)
h_2^(dec) = LSTM(c, E[1], h_1^(dec))
Output: P(y_2|y_1,c) = [0.02, 0.03, 0.93, 0.02]  # Predicts "feliz" (token 2)

Input: y_2 = "feliz" (token 2)
h_3^(dec) = LSTM(c, E[2], h_2^(dec))
Output: P(y_3|y_1,y_2,c) = [0.01, 0.02, 0.03, 0.94]  # Predicts <EOS> (token 3)

Loss calculation (assuming these are the correct predictions): $\mathcal{L} = -\log(0.92) - \log(0.93) - \log(0.94) \approx 0.24$

Why this step? We sum negative log-probs of the correct tokens. Lower loss means higher confidence in correct sequence.

Common Decoding Strategies

1. Greedy Decoding

Pick the highest-probability token at each step: $y_t = \arg\max_{v} P(v \mid y_{<t}, \mathbf{c})$

Fast but suboptimal: Doesn't consider future consequences. Picking the best word now might lead to a dead-end later.

2. Beam Search

Maintain top-$k$ candidate sequences at each step. Expand, keep top-$k$ again.

Example with beam width $k=2$:

Step 1: <SOS> → {"Estoy" (0.9), "Yo" (0.7)}

Step 2 (expand both):
  "Estoy" → {"Estoy feliz" (0.9×0.8=0.72), "Estoy contento" (0.9×0.6=0.54)}
  "Yo" → {"Yo soy" (0.7×0.5=0.35), "Yo estoy" (0.7×0.4=0.28)}
  
Keep top 2: {"Estoy feliz" (0.72), "Estoy contento" (0.54)}

Why beam search? Balances exploration and efficiency. Considers multiple paths without exhaustive search.

Special Tokens and the Mechanism

Problem: How does the decoder know when to stop?

Solution: Add a special end-of-sequence (EOS) token to the vocabulary. Train the model to predict <EOS> when the output is complete.

Training data:
Input:  "Hello"
Target: "<SOS> Hola <EOS>"

At inference, we stop generating when the model outputs <EOS> or we hit a max length (safety).

Why <SOS>? The decoder needs an initial input. <SOS> (start-of-sequence) signals the beginning.

The Information Bottleneck Problem

Why it feels right: The LSTM's hidden state is designed to carry forward information.

The reality: A fixed-size vector $\mathbf{c}$ (e.g., 512 dimensions) must encode everything about a potentially long input (e.g., 50 tokens, each with rich meaning). For long sequences, early information gets forgotten—the LSTM's memory is limited.

The fix: Attention mechanisms (next topic). Instead of one fixed $\mathbf{c}$, the decoder computes a different context vector at each decoding step by "looking back" at all encoder hidden states.

Steel-man the mistake: For short sequences (5-10 tokens), the bottleneck isn't severe. The original seq2seq paper (Sutskever et al., 2014) got good results on moderate-length sentences. The problem becomes critical for documents, paragraphs, or nuanced context.

Bidirectional Encoder Improvement

Limitation: A unidirectional encoder at token $t$ has only seen $x_1, \dots, x_t$, not future tokens.

Enhancement: Use a bidirectional RNN/LSTM as encoder: $\overrightarrow{h}_t = \text{LSTM}_{\text{forward}}(x_t, \overrightarrow{h}_{t-1})$ $\overleftarrow{h}_t = \text{LSTM}_{\text{backward}}(x_t, \overleftarrow{h}_{t+1})$

Concatenate: $h_t^{(enc)} = [\overrightarrow{h}_t; \overleftarrow{h}_t]$

The context vector becomes: $\mathbf{c} = [h_{T_x}^{\text{forward}}; h_1^{\text{backward}}]$

Why this helps: The encoder now has information about the entire sequence at every position. The word "bank" can be understood differently if followed by "river" vs. "money".

Input: "The cat sat on the mat. It was very comfortable."
Target: "Cat on mat."

Encoder

Tokenize input: ["The", "cat", "sat", "on", "the", "mat", ".", "It", "was", "very", "comfortable", "."]
Process through bidirectional LSTM → $\mathbf{c} \in \mathbb{R}^{512}$

Decoder

t=1: Input <SOS>, output "Cat" (prob 0.85)
t=2: Input "Cat", output "on" (prob 0.78)
t=3: Input "on", output "mat" (prob 0.82)
t=4: Input "mat", output "." (prob 0.91)
t=5: Input ".", output <EOS> (prob 0.95)

Loss: $-\log(0.85) - \log(0.78) - \log(0.82) - \log(0.91) - \log(0.95) \approx 0.77$

Why each step?

• t=1: Decoder learns to pick the most salient subject.
• t=2-3: Learns to compress "sat on the" → "on".
• t=4: Punctuation is part of well-formed output.
• t=5: Termination signal.

When to Use Seq2Seq

Not ideal for: Tasks where input/output alignment is known (e.g., POS tagging—here, each input word gets one tag; simpler models suffice).

So you do it in two steps:

1. Listen carefully to the whole English sentence and remember the idea in your head (that's the encoder making the context vector).
2. Say it in Spanish, one word at a time, using what you remembered (that's the decoder).

The "context vector" is like a mental note you made: "They're talking about a happy cat on a mat." Then you speak Spanish words that match that mental note, not the exact English words. That's why the lengths can be different—you're translating ideas, not just words!

Or simply: "Understand, Remember, Speak" (Encoder understands, Context remembers, Decoder speaks).

Connections

• 3.5.05-LSTM-networks – Encoder/decoder typically use LSTMs to handle long sequences
• 3.5.06-GRU-networks – Alternative to LSTM in seq2seq, faster training
• 3.5.08-Attention-mechanism – Solves the context bottleneck by letting decoder "attend" to all encoder states
• 3.5.09-Transformers – Replace RNNs with self-attention, modern seq2seq backbone
• 3.4.03-Backpropagation-through-time – How gradients flow through seq2seq RNNs
• 4.2.01-Beam-search – Decoding strategy for seq2seq
• 2.3.02-Cross-entropy-loss – Loss function for training seq2seq models
• 3.5.01-Recurrent-neural-networks – Foundation of encoder/decoder architecture

#flashcards/ai-ml

What are the two main components of a seq2seq model? :: Encoder (processes input into context vector) and Decoder (generates output from context)

Concept Map

Hinglish (regional understanding)