Sequence-to-sequence models: Visual walkthrough

We assume you have met recurrent neural networks (a box that reads one word, remembers, then reads the next). Everything else we earn from zero.

Definition Notation set up before we start

A few names recur on every step — pin them down now so no symbol appears unexplained.

$T_{x}$ ::: the number of words in the input sentence (here $T_{x} = 3$ : "I", "am", "happy").
$T_{y}$ ::: the number of words in the output sentence.
$x_{t}$ ::: the embedding vector (a fixed list of numbers) for the input word at position $t$ .
$y_{t}$ ::: the output word (a discrete token from the vocabulary) the decoder produces at position $t$ ; $y_{t - 1}$ is simply the one it produced the step before.
$E [\cdot]$ ::: the embedding lookup — it takes a discrete word token and returns its vector. $E ["am"] = x_{2}$ . We'll need it again to turn output tokens into vectors.
<SOS> ::: a special start-of-sequence token — an extra "word" added to the vocabulary that means "begin generating now". It is the decoder's very first input.
<EOS> ::: a special end-of-sequence token that means "the sentence is finished". The decoder is trained to emit it when done.

Step 1 — A sentence is a line of boxes

WHY. We need numbers because the only thing a neural network can multiply and add is numbers. The word "am" always maps to the same vector $x_{2} = E ["am"]$ , so the network can learn what "am" means once and reuse it everywhere.

PICTURE. Look at figure s01. Three coloured dots sit on a track, left to right — that is time. Each dot has a little arrow beneath it: the embedding vector for that word. Nothing has been "understood" yet; these are just three separate arrows in a row.

Step 2 — The encoder walks the line and remembers

$new memory h_{t}^{(e n c)} = LSTM (word now x_{t}, old memory h_{t - 1}^{(e n c)})$

Read it left to right: new memory = a fixed recipe applied to (the current word, the memory I had a moment ago). The recipe is an LSTM (or a GRU) — a box built to keep information alive across many steps without it fading too fast.

WHY an LSTM and not a plain RNN? A plain RNN's memory decays quickly, so by word 20 it has half-forgotten word 1. The LSTM adds a "keep this" gate that lets important facts survive the whole walk. We choose it precisely because we must carry information across the entire sentence.

WHY feed the old memory back in? Because meaning is contextual. "Bank" after "river" differs from "bank" after "money". Feeding $h_{t - 1}$ in means each new reading happens in light of everything already read.

PICTURE. In figure s02 the encoder box moves rightward. The blue arrow entering the first box from the left is the blank start $h_{0} = 0$ ; the yellow arrow coming up is the current word $x_{t}$ ; the blue arrow leaving on the right is the updated memory $h_{t}$ . Notice the memory arrow never breaks — that is the chain of understanding.

Step 3 — Squeeze the whole sentence into one dot

$c = h_{T_{x}}^{(e n c)}$

$c$ (left) is defined to equal the encoder's last hidden state (right). No new computation — just a promotion. This is the "thought" the parent note called it.

WHY the last state and not, say, the first? Because only the last state has walked past all the words. The first state has seen just one. Information in an LSTM flows forward, so "latest = most complete".

PICTURE. Figure s03 shows the three memory arrows collapsing into a single glowing dot on the right, labelled $c$ . Three words in → one point of meaning. (The faint red halo hints at the flaw we return to in Step 8: a lot got crammed into one dot.)

Step 4 — The decoder unfolds the dot, one word at a time

$h_{t}^{(d ec)} = LSTM (the meaning c, vector of the word I just said E [y_{t - 1}], my running memory h_{t - 1}^{(d ec)})$

Each ingredient: the fixed meaning $c$ (so we never forget what we're translating), the embedded previous output token $E [y_{t - 1}]$ (the token turned back into a vector, so the next word is grammatically consistent), and the decoder's own memory $h_{t - 1}^{(d ec)}$ .

WHY feed back the previous word? Language is a chain: "am" is likely after "I". If the decoder ignored its own last word, it might say "I I happy". Feeding $E [y_{t - 1}]$ back keeps the output a coherent sentence.

PICTURE. Figure s04 mirrors s02 but flowing outward: the green dot $c$ seeds the first memory and feeds every decoder box; the <SOS> token starts it (embedded into a vector); each produced word (yellow) is embedded and loops back down into the next box. Compare the shapes — encoder folds in, decoder unfolds out.

Step 5 — Turn a memory vector into a word choice (softmax)

$P (y_{t} ∣ y_{< t}, c) = softmax (one raw score per word W_{o} h_{t}^{(d ec)} + b_{o})$

$W_{o}$ ::: the output weight matrix; multiplying it by the memory produces one raw score per vocabulary word.
$b_{o}$ ::: the output bias — a fixed learned vector added to every score, letting the model shift a word's baseline likelihood up or down regardless of the memory. One entry per vocabulary word.
$W_{o} h_{t}^{(d ec)} + b_{o}$ ::: together give a raw number ("logit") for each of the $∣ V ∣$ words — bigger means "more likely".
$softmax$ ::: squashes those raw numbers into positive fractions that add to 1.

The softmax formula, expanded, just says "each word's share = its exponential divided by the sum of all exponentials": $softmax (z)_{v} = \frac{e x p ( z _{v} )}{\sum _{u \in V} e x p ( z _{u} )}$

WHY softmax and not just "pick the biggest raw number"? We want a probability distribution — for training we need to say "the true word should get 0.92 probability", and for beam search we need to multiply probabilities of whole sentences. Raw scores can't be multiplied meaningfully; probabilities can. The exponential guarantees every value is positive, and the division guarantees they sum to 1.

PICTURE. Figure s05: the memory arrow enters a bar chart. Four raw scores become four bars whose heights are probabilities summing to 1. The tallest bar — "Estoy" at 0.92 — is the chosen word.

Step 6 — Score the whole run, then train it (cross-entropy)

Whole-sentence probability (a product because we generate word by word): $P (y_{1}, \dots, y_{T_{y}} ∣ x) = \prod_{t = 1}^{T_{y}} P (y_{t} ∣ y_{< t}, c)$

Loss (turn the product into a sum with a logarithm, then flip the sign): $L = - \sum_{t = 1}^{T_{y}} lo g P (y_{t}^{*} ∣ y_{< t}^{*}, c)$

$y_{t}^{*}$ ::: the correct word at position $t$ (the star = ground truth).
$- lo g$ ::: turns "probability 1" into loss 0, and "probability near 0" into a huge penalty.
$\sum$ ::: we add the per-word penalties across the sentence.

WHY the logarithm? Multiplying many small probabilities gives a tiny number the computer can't handle. $lo g$ turns the product into a sum, which is stable and easy to differentiate for backpropagation through time. WHY the minus? Maximising probability = minimising negative log-probability; optimisers are built to minimise.

Worked numbers. With correct-word probabilities $0.92, 0.93, 0.94$ (using natural logarithm): $L = - lo g (0.92) - lo g (0.93) - lo g (0.94) \approx 0.213$

PICTURE. Figure s06 stacks three probability bars for the correct words and shows each contributing a small red penalty $- lo g p$ ; the penalties add up to the total loss. Confident-and-correct → short red bars → low loss.

Step 7 — Choosing words at inference: greedy vs. beam

Greedy pick: $y_{t} = ar g max_{v} P (v ∣ y_{< t}, c)$ "Take the word $v$ with the largest probability."

Beam score (average log-prob so long and short sentences compete fairly): $score (y_{1 : t}) = \frac{1}{t} \sum_{i = 1}^{t} lo g P (y_{i} ∣ y_{< i}, c)$

WHY beam over greedy? Greedy is a person who always takes the nearest turn and gets lost. Beam keeps a few routes open and picks the best whole path. The $\frac{1}{t}$ length normalisation stops the model from cheating by choosing very short sentences (fewer factors → higher raw product).

PICTURE. Figure s07 is a small tree with beam width $k = 2$ . Two branches survive at each level (solid), losers are dashed. The kept branch "Estoy feliz" (product $0.72$ ) beats "Estoy contento" ( $0.54$ ); "Yo…" branches die early.

Recall Check the beam arithmetic

"Estoy" 0.9 → "feliz" 0.8 gives ::: $0.9 \times 0.8 = 0.72$ , which beats "Estoy contento" $0.9 \times 0.6 = 0.54$ and both "Yo" branches ( $0.35$ , $0.28$ ).

Step 8 — The degenerate case: when one dot isn't enough

WHY it must fail eventually. $c$ has a fixed size (say 512 numbers) no matter how long the input is. A short sentence half-fills it comfortably; a long paragraph overflows it, and the LSTM's forward-only memory forgets the opening words by the time it reaches the end. Translation quality drops sharply as length grows.

Edge cases to keep straight:

Empty / one-word input ( $T_{x} = 1$ ): $c = h_{1}$ ; works perfectly, the bottleneck is irrelevant.
Very long input ( $T_{x}$ large): $c$ saturates, early words vanish — quality collapses.
Never emitting <EOS>: the decoder could loop forever, so we always cap at a maximum length as a safety stop.

The fix (next topic). Instead of one frozen $c$ , let the decoder build a fresh context at every output step by looking back at all encoder states — the attention mechanism, which powers Transformers.

PICTURE. Figure s08 plots translation quality against sentence length: a flat high line for the attention model versus a curve that rises then droops for the single-vector seq2seq — the droop is the bottleneck biting.

The one-picture summary

Figure s09 puts the whole pipeline on one canvas: words flow right into the blue encoder (starting from blank memory $h_{0} = 0$ ), fold into the green meaning-dot $c$ , which seeds and feeds the yellow decoder that unfolds outward into a new sentence, closed by <EOS>. The red halo on $c$ is the reminder of Step 8.

Recall Feynman retelling — say it like a story

A reader (encoder) starts with a blank memory ( $h_{0} = 0$ ) and walks along the input sentence, one word at a time, keeping a running memory that it feeds forward so each word is understood in context. When it reaches the end, its final memory becomes a single "thought" $c$ . A writer (decoder) is seeded with that thought (its $h_{0}^{(d ec)} = c$ ), starts with <SOS>, embeds it into a vector, and produces one word; then it embeds its own last word plus keeps the thought to produce the next word, and keeps going until it writes <EOS>. To pick each word it turns its memory into scores, adds a bias, and softmaxes them into probabilities. While learning, we give it the true previous word (teacher forcing) and punish it by the negative log-probability of the correct word — that's cross-entropy. At test time we can grab the best word each step (greedy) or keep a few sentences alive (beam search) to find the best whole sentence. The one weakness: everything must pass through that single thought-dot, which overflows on long inputs — and that is exactly the door that attention opens, leading us to Transformers.

Links: parent Sequence-to-sequence models · prerequisites 3.5.01-Recurrent-neural-networks, 3.5.05-LSTM-networks, 3.5.06-GRU-networks · training 3.4.03-Backpropagation-through-time, 2.3.02-Cross-entropy-loss · decoding 4.2.01-Beam-search · what fixes the bottleneck 3.5.08-Attention-mechanism, 3.5.09-Transformers · Hinglish version