Encoder-decoder architecture: Visual walkthrough

Step 1 — What a "sequence" even is, and why one vector can't hold it yet

WHAT. A sequence is just an ordered list of things read left to right. Our input is $x = (x_{1}, x_{2}, \dots, x_{T_{x}}) .$ Let me name every piece:

$x_{t}$ ::: the item at position $t$ — for us, one word turned into numbers.
$t$ ::: the position counter, ticking $1, 2, 3, \dots$
$T_{x}$ ::: the total number of input items (the length of the input). The little " $x$ " reminds us it is the input length.

WHY we care. The order matters. "Dog bites man" and "man bites dog" use identical words but mean opposite things. A machine that ignores order would confuse them. So we need something that reads the words in sequence and remembers what came before.

PICTURE. Look at the row of boxes: each box is one word $x_{t}$ . The red arrow shows the single direction we read — earlier words influence later understanding, never the reverse (for now).

Step 2 — The memory cell: how the encoder "reads and remembers"

WHAT. We introduce a running memory that updates once per word. Call it $h_{t}^{e n c}$ — the encoder's hidden state after reading word $t$ : $h_{t}^{e n c} = f_{e n c} (new word in x_{t}, memory so far h_{t - 1}^{e n c}) .$

Term by term:

$h_{t}^{e n c}$ ::: the memory after seeing word $t$ (a list of numbers).
$h_{t - 1}^{e n c}$ ::: the memory before word $t$ — everything understood up to now.
$x_{t}$ ::: the fresh word being added.
$f_{e n c}$ ::: the update rule (an LSTM or GRU cell). It blends old memory with the new word.

WHY this recursion and not a plain sum? A sum would forget order — $x_{1} + x_{2} = x_{2} + x_{1}$ . Feeding the previous memory back in makes each state depend on the path taken to reach it. That is exactly what an RNN gives us: order-sensitive memory. We choose recursion because the question "what did the whole prefix mean?" can only be answered by folding words in one at a time.

PICTURE. The red arrow is the feedback loop: yesterday's memory flows into today's cell. Every box hands its state rightward, so the last box has (in principle) touched every word.

Step 3 — Squeezing everything into one context vector $c$

WHAT. After the last word $x_{T_{x}}$ , we freeze the final memory and call it the context vector: $c = h_{T_{x}}^{e n c} .$

$c$ ::: one fixed-size list of numbers meant to summarise the entire input.
$h_{T_{x}}^{e n c}$ ::: the memory after the last word — the end of the chain from Step 2.

WHY the last state? Because of the recursion, $h_{T_{x}}^{e n c}$ was built from $h_{T_{x} - 1}^{e n c}$ , which was built from $h_{T_{x} - 2}^{e n c}$ , and so on back to $x_{1}$ . So the last state is the only single point that "has heard" the whole sentence.

PICTURE. All arrows funnel into one red dot. That dot is $c$ — the bottleneck through which all meaning must pass.

Step 4 — The decoder starts: handing over the baton

WHAT. The decoder is a second memory chain. It begins its life as the context: $h_{0}^{d ec} = c, y_{0} = ⟨ START ⟩ .$

$h_{0}^{d ec}$ ::: the decoder's starting memory — copied from $c$ .
$y_{0}$ ::: a fake "word zero," the $⟨ START ⟩$ signal telling the decoder "begin producing now."

WHY start from $c$ ? The decoder has no other window into the input. Its only knowledge of "I love AI" is the summary handed over. Initialising its memory to $c$ literally injects that knowledge into the generation process.

PICTURE. The red dot from Step 3 flows across the divide and becomes the first block of the output chain. The baton is passed.

Step 5 — Generating one word at a time (autoregression)

WHAT. The decoder updates its memory using the word it just spoke. But careful: $y_{t - 1}$ is a discrete word (like "J'aime") — and a cell only eats numbers. So, exactly as we did for the input in Step 1, we first turn that word into an embedding vector $emb (y_{t - 1})$ before feeding it in: $h_{t}^{d ec} = f_{d ec} (last word I said, turned into numbers emb (y_{t - 1}), my memory h_{t - 1}^{d ec}) .$

$y_{t - 1}$ ::: the previously generated output word (a discrete token).
$emb (y_{t - 1})$ ::: that word mapped into a continuous vector — the decoder's analogue of $x_{t} = emb (word)$ from Step 1.
$h_{t - 1}^{d ec}$ ::: the decoder's memory before this step.
$f_{d ec}$ ::: the decoder's update rule (another LSTM/GRU cell).

WHY embed it first? A word like "J'aime" is just an index into a dictionary; the cell cannot do arithmetic on an index. The embedding lookup gives it coordinates in meaning space — the same trick that let the encoder read words. Same problem, same fix, both ends of the machine.

WHY feed the previous word back at all? To stay coherent. If it just said "J'aime", the next word should follow that, not float free. This "use my own last output as next input" loop is called autoregression.

PICTURE. Each output box loops its word up and into the next box (red loop) — mirror image of the encoder, but now the arrows produce words instead of consuming them.

Step 6 — From memory to an actual word: the softmax

WHAT. A hidden state is numbers, not a word. We turn it into a probability for every word in the vocabulary $V$ : $z = W_{o} h_{t}^{d ec} + b_{o}, P (y_{t} = v ∣ y_{< t}, x) = \frac{e ^{z_{v}}}{\sum _{u \in V} e ^{z_{u}}} .$ Term by term:

$W_{o}, b_{o}$ ::: a weight matrix and bias that map the memory to one raw score per word.
$z$ ::: the vector of raw scores ("logits"), one entry $z_{v}$ per candidate word $v$ .
$z_{v}$ ::: the raw score of word $v$ ; bigger = more favoured.
$e^{z_{v}}$ ::: exponential — makes every score positive and blows up gaps so the winner stands out.
the denominator $\sum_{u} e^{z_{u}}$ ::: adds up all the exponentials so the result divides to $1$ .
$y_{< t}$ ::: shorthand for "all words spoken before position $t$ ."

WHY exponentiate and divide? We need genuine probabilities: all non-negative, summing to $1$ , so the model can pick a word. Exponentials guarantee positivity; dividing by their total guarantees they sum to one. That is exactly what softmax does — it is the tool that answers "given these scores, how likely is each word?" One of the words competing for probability here is $⟨ END ⟩$ from Step 5 — that is how "stop now" competes fairly against every real word.

PICTURE. Tall red bar = the chosen word ("am"); the transform squashes a spiky score vector into a clean probability bar chart that sums to 1.

Step 7 — Why the training loss looks the way it does

WHAT. We want the model to make the true sentence $y^{*}$ likely. Its probability factorises by the chain rule of probability: $P (y^{*} ∣ x) = \prod_{t = 1}^{T_{y}} P (y_{t}^{*} ∣ y_{< t}^{*}, x) .$

$\prod$ ::: multiply the per-word probabilities together (they chain: each word's chance depends on the ones before).
$y_{t}^{*}$ ::: the correct (ground-truth) word at position $t$ ; the star means "the target."
$T_{y}$ ::: length of the output sequence.

Multiplying many small numbers underflows to zero on a computer, so we take a logarithm — which turns products into sums and is easier and stabler: $lo g P (y^{*} ∣ x) = \sum_{t = 1}^{T_{y}} lo g P (y_{t}^{*} ∣ y_{< t}^{*}, x) .$ Maximising probability = minimising its negative, averaged over $N$ examples: $L (θ) = - \frac{1}{N} \sum_{i = 1}^{N} \sum_{t = 1}^{T_{y}^{(i)}} lo g P_{θ} (y_{t}^{(i)} ∣ y_{< t}^{(i)}, x^{(i)}) .$

$θ$ ::: every weight in encoder and decoder — what training adjusts.
the minus sign ::: converts "maximise likelihood" into "minimise loss."

WHY the log, concretely? Multiplying $0.001 \times 0.002 \times \dots$ 30 times gives $\sim 1 0^{- 90}$ — the computer rounds it to $0$ and learning dies. Adding $lo g$ -values ( $- 3 - 2.7 - \dots$ ) never underflows. The gradients that update $θ$ flow backward through time via BPTT.

PICTURE. Two arithmetic ladders side by side: the product ladder plunges toward $0$ ; the log-sum ladder walks steadily downhill — same information, safe numbers. The red line is the safe one.

Step 8 — The degenerate case: when one vector is too small

WHAT. Everything above squeezes an arbitrarily long input through the single fixed vector $c$ . What happens as $T_{x}$ grows?

WHY it must break. The key is not a precise bit-count (the slots of $c$ are real numbers, whose usable information is limited by noise and training, not by their 32-bit storage — so raw bit arithmetic would mislead us). The honest argument is about capacity that does not grow: $dim (c) = 512 (fixed), information to store \propto T_{x} (grows without limit) .$

$dim (c)$ ::: the number of slots in the context vector — a constant chosen once (say 512).
$T_{x}$ ::: the input length — free to grow to 20, 50, 200 words.

Because $c$ has a fixed number of slots but the input it must summarise keeps growing, at some length the vector simply runs out of room. Details — especially early words, which have passed through the most update steps — get overwritten. This is the information bottleneck. Empirically, plain seq2seq accuracy falls off sharply once sentences pass roughly 20 words, exactly where a single vector starts to saturate.

PICTURE. A wide funnel (the long input) forced through a narrow red neck (the fixed-size $c$ ) — geometry that makes the loss inevitable. Short inputs pass; long ones spill over the sides.

The one-picture summary

Left half reads words into a chain of memories that funnel into the red context dot $c$ ; right half unfurls that dot back into words, each fed forward (after being re-embedded), each turned into a word by softmax, until $⟨ END ⟩$ is chosen. One drawing, the whole derivation.

Recall Feynman retelling — say it back in plain words

First I take a sentence and read it one word at a time, keeping a little scratchpad of "what I understand so far." Each new word updates the scratchpad. When I reach the end, I freeze the scratchpad into one fixed-size summary — the context vector $c$ . Then I start a second scratchpad set to that summary and begin speaking. Each word I say, I turn back into numbers (an embedding lookup) and feed to myself so my next word stays coherent. To choose a word I score every word in my vocabulary, exponentiate the scores so the best ones dominate, and normalise so they're real probabilities (softmax) — then pick. I keep going until I pick the special $⟨ END ⟩$ token, which means "the sentence is finished," or until I hit the length cap $T_{m a x}$ . To learn, I make the true sentence probable. Since a sentence's probability is a product of per-word probabilities, and products of tiny numbers vanish, I take logs and add instead — maximising that sum, which equals minimising the negative-log-likelihood loss. During training I feed myself the correct previous word (teacher forcing) so I learn fast. The catch: cramming an ever-longer sentence through one fixed-size vector must eventually lose information, because the vector's capacity never grows while the input does. That bottleneck is exactly why attention was born.

Recall Quick self-test

Why can't the encoder use a plain sum of word vectors? ::: A sum ignores order ( $x_{1} + x_{2} = x_{2} + x_{1}$ ); the recurrence $h_{t} = f (x_{t}, h_{t - 1})$ makes memory depend on the path/order. What single quantity bridges variable-length input and output? ::: The fixed-size context vector $c = h_{T_{x}}^{e n c}$ . Before feeding $y_{t - 1}$ back into the decoder, what must you do? ::: Map the discrete word into a continuous vector via an embedding lookup, $emb (y_{t - 1})$ — just like $x_{t}$ in the encoder. How does generation know when to stop? ::: When the chosen token is the special $⟨ END ⟩$ symbol, or when the length cap $T_{m a x}$ is reached. Why take the log of the likelihood? ::: Products of many small probabilities underflow to 0; logs turn products into stable sums. Why does softmax exponentiate? ::: To make scores positive and amplify the winner, then divide so they sum to 1 — a valid probability distribution. What breaks for long inputs and what fixes it? ::: The context vector's capacity is fixed while input length grows, so it saturates (information bottleneck); letting the decoder read all encoder hidden states (the attention mechanism) fixes it.