Worked examples — Encoder-decoder architecture

This is a companion drill page for Encoder-decoder architecture. The parent note built the machinery: an encoder squeezes the input sequence into a single context vector $\mathbf{c}$, and a decoder unrolls that vector into an output sequence, one token at a time. Here we do nothing but crank the handle — we push real numbers through the formulas and check every corner case, so that when you meet these on an exam or in code you have already seen the exact scenario.

Before the examples, one promise: every symbol used here was defined in the parent, and we re-anchor each as it appears. If a number looks like it came from nowhere, that is a bug — hunt it.

---

The scenario matrix

Think of the encoder-decoder as a machine with knobs. Each knob can be at a "normal" setting, a "zero/edge" setting, or a "degenerate/broken" setting. The table below lists every case class this topic can throw at you — 10 cells labelled A–J. The 8 worked examples underneath are each tagged with the cell(s) they cover, and together they fill all 10 cells (some examples cover two cells at once, which is why 8 examples suffice for 10 cells).

Cell	Case class	What is weird about it	Covered by
A	Normal forward pass	Ordinary lengths, everything defined	Ex 1
B	Length mismatch $T_x\ne T_y$	Input 3 tokens → output 4 tokens	Ex 1, Ex 2
C	Softmax numerics	Turning raw scores into probabilities	Ex 3
D	Log-likelihood loss	Product → sum via logs	Ex 4
E	Teacher forcing vs free-run	Same weights, different fed inputs	Ex 5
F	Greedy vs beam (global search)	Local best ≠ global best	Ex 6
G	Degenerate input: empty / single token	$T_x=0$ or $T_x=1$	Ex 7
H	Limiting behaviour: bottleneck	$T_x\to$ large, fixed $\mathbf{c}$	Ex 8
I	Real-world word problem	Translation with counts	Ex 2
J	Exam twist: perplexity	Loss reported as a "surprise" number	Ex 4

Prerequisites you may want open: 3.5.01-Recurrent-neural-networks, 3.5.03-LSTM-long-short-term-memory, 3.4.06-Beam-search, 3.5.09-Attention-mechanism.

---

Example 1 — Cell A, B: one full forward pass, by hand

Forecast: Guess the sign of $\mathbf{c}$. The last token is $+0.5$, but $x_2=-2$ is a big negative pull. Which wins?

1. Compute $h_1^{enc}=\tanh (1\cdot 1+0.5\cdot 0+0)=\tanh (1)$. Why this step? The parent's recursion $h_t^{enc}=f_{enc}(x_t,h_{t-1}^{enc})$ says each state depends only on the current token and the previous state. With $h_0=0$ the "memory" term vanishes on step 1. $\tanh (1)\approx 0.7616$.

2. Compute $h_2^{enc}=\tanh (1\cdot (-2)+0.5\cdot 0.7616)=\tanh (-1.6192)$. Why this step? Now the memory term $u{h}_1=0.5\cdot 0.7616=0.3808$ carries what "token 1" left behind. The negative token dominates. $\tanh (-1.6192)\approx -0.9243$.

3. Compute $h_3^{enc}=\tanh (1\cdot 0.5+0.5\cdot (-0.9243))=\tanh (0.0378)$. Why this step? The final hidden state is our context vector — the parent says $\mathbf{c}=h_{T_x}^{enc}$, "the state that has seen the whole sequence." $\tanh (0.0378)\approx 0.0378$.

4. Answer: $\mathbf{c}\approx 0.0378$ (barely positive).

Verify: The negative $x_2$ pushed $h_2$ near $-0.92$; the positive $x_3$ plus that decayed memory nearly cancelled to give $\approx 0.038$. The near-zero result is the tug-of-war our forecast predicted — the recent positive token barely edged out the decayed negative memory. Range check: $\tanh$ always outputs in $(-1,1)$, and $0.0378$ is safely inside. ✓

How the figure reinforces the numbers: the three yellow dots are literally $h_1,h_2,h_3$ from steps 1–3 plotted against time. Notice the red arrow at step 2 — that is $x_2=-2$ dragging the state down to $-0.92$; then the green arrow at step 3 shows $x_3=+0.5$ hauling it back up across the zero line to $+0.038$. The slope of each segment is exactly the "pull" of that token, so the near-flat final segment visually confirms our tug-of-war ended in a near-tie.

---

Example 2 — Cell B, I: length mismatch word problem

Forecast: Guess how many times $f_{dec}$ fires. Is it 3? 4? 5?

1. Encoder fires $T_x=3$ times, producing $\mathbf{c}=h_3^{enc}$. Why this step? One encoder step per input token — the input length sets the encoder's loop count.

2. Decoder fires once per output token, plus one final step to emit $⟨\text{END}⟩$. Output has $T_y=4$ real tokens, so decoder runs $4+1=5$ times. Why this step? The decoder is autoregressive — it keeps generating until it produces the stop symbol. The stop symbol is itself a token requiring a step.

3. Why the mismatch is fine: the context vector $\mathbf{c}$ is a fixed-size bridge — the whole point from the parent's intuition. Once the input collapses to $\mathbf{c}$, the decoder can generate any number of tokens.

4. Answer: encoder steps $=3$, decoder steps $=5$, total recurrent applications $=8$.

Verify: $3+5=8$. If input and output were both length 3, we'd get $3+4=7$; the extra decoder step here is the $⟨\text{END}⟩$ emission plus the 4th French token. Counting is consistent. ✓

---

Example 3 — Cell C: softmax turned into real probabilities

Forecast: "am" has the biggest logit — will its probability be above or below 60%?

1. Exponentiate each logit. $e^{2.0}=7.389,e^{1.0}=2.718,e^{0.1}=1.105$. Why this step? Softmax is $P(y_t=v)=\frac{e^{z_v}}{{\sum }_u{e}^{z_u}}$. The exponential is chosen (not, say, just dividing raw scores) because it is always positive — you cannot have a negative probability — and it magnifies gaps between scores.

2. Sum them: $7.389+2.718+1.105=11.212$. Why this step? The denominator is the normalizer that forces ${\sum }_{v}P(y_t=v)=1$.

3. Divide: $P(\text{"am"})=7.389/11.212\approx 0.659$. Why this step? This is the softmax ratio for the "am" slot.

4. Answer: $P(\text{"am"})\approx 0.659$, i.e. about $65.9\%$.

Verify: The other two are $2.718/11.212\approx 0.242$ and $1.105/11.212\approx 0.099$. Sum: $0.659+0.242+0.099=1.000$. A valid distribution. ✓ Above our 60% forecast — exponentiation exaggerated the lead.

How the figure reinforces the numbers: the blue bars are the raw logits $z=(2.0,1.0,0.1)$ and the yellow bars are the probabilities from step 3. Note that the logit gap between "am" and "is" is only $1.0$, yet the yellow probability gap ($0.659$ vs $0.242$) is far larger — that visual stretching is the exponential "magnifying gaps" that step 1 warned about. The three yellow bars are also drawn to sum to the height 1, which is the normalization from step 2 made visible.

---

Example 4 — Cell D, J: log-likelihood loss and perplexity (exam twist)

Forecast: Perplexity is "the effective number of equally-likely choices the model was juggling." Guess: closer to 2, or closer to 10?

1. Joint probability by chain rule: $P({\mathbf{y}}^{\ast })=0.5\times 0.25\times 0.8=0.1$. Why this step? The parent decomposes $P({\mathbf{y}}^{\ast }\mid \mathbf{x})={\prod }_{t}P(y_t^{\ast }\mid y_{<t}^{\ast },\mathbf{x})$ — each token's probability conditioned on the ones before.

2. Log-likelihood: $\log P({\mathbf{y}}^{\ast })=\ln 0.5+\ln 0.25+\ln 0.8$. Why this step? We take logs (natural log here) to turn the product into a sum — this avoids tiny numbers underflowing to zero and matches the parent's $\mathcal{L}={\sum }_t\log P(\cdot )$. $=-0.6931-1.3863-0.2231=-2.3026$.

3. Mean cross-entropy loss: $\mathcal{L}=-\frac{1}{T_y}{\sum }_t\log P=-\frac{1}{3}(-2.3026)=0.7675$. Why this step? The parent's objective divides by the number of tokens so sequences of different length are comparable.

4. Perplexity: $\text{PPL}=e^{\mathcal{L}}=e^{0.7675}\approx 2.154$. Why this step? Perplexity is the exam-favourite reformatting of loss: $\text{PPL}=\exp (\text{mean NLL})$. It reads as "on average the model behaved as if choosing among $\approx 2.15$ equally likely tokens."

Verify: Sanity check via geometric mean: $\text{PPL}=(0.5\cdot 0.25\cdot 0.8{)}^{-1/3}=0.1^{-1/3}=10^{1/3}\approx 2.154$. Matches step 4. Our forecast "closer to 2" was right. ✓

---

Example 5 — Cell E: teacher forcing vs free running, same weights

Forecast: Will the two hidden states differ by a little or a lot?

1. Teacher forcing feeds $y_1^{\ast }=1.0$. $h_2^{TF}=\tanh (1\cdot 1.0+0.5\cdot 0.2)=\tanh (1.1)\approx 0.8005$. Why this step? The parent's teacher forcing rule: at training time we feed the correct previous token regardless of what the model said.

2. Free running feeds the model's own ${\hat{y}}_1=-0.5$. $h_2^{FR}=\tanh (1\cdot (-0.5)+0.5\cdot 0.2)=\tanh (-0.4)\approx -0.3799$. Why this step? At test time (or in scheduled sampling) the decoder eats its own output — this is where exposure bias lives.

3. Gap: $0.8005-(-0.3799)=1.1804$ — nearly the whole $(-1,1)$ range apart.

4. Answer: one wrong token flipped the hidden state from strongly positive to negative.

Verify: Both outputs lie in $(-1,1)$ ✓. The huge gap ($1.18$ out of a max span of $2$) is a numerical demonstration of why early mistakes compound — exactly the parent's Mistake 2. ✓

---

Example 6 — Cell F: greedy decoding is NOT optimal

Forecast: Greedy grabs "The" first (it's more likely). Does greedy also win the whole sequence?

1. Greedy step 1: pick the higher first-token probability → "The" ($0.5>0.4$). Why this step? Greedy = argmax at each step, ignoring the future.

2. Greedy full path: "The" → "dog", total $=0.5\times 0.3=0.15$. Why this step? Sequence probability multiplies step probabilities (chain rule again).

3. Alternative path: "A" → "fox", total $=0.4\times 0.5=0.20$. Why this step? Beam search keeps this hypothesis alive even though "A" lost round 1.

4. Answer: global best $=0.20$ ("A fox") beats greedy $=0.15$ ("The dog"). Greedy is sub-optimal.

Verify: $0.20>0.15$ ✓. The locally-losing first move led to the globally-winning sequence — the parent's Mistake 3, made concrete. Beam width $\ge 2$ would have caught it. ✓

---

Example 7 — Cell G: degenerate inputs (empty and single token)

Forecast: For the empty input, is $\mathbf{c}$ undefined, or something specific?

1. Empty input: no encoder step runs, so $\mathbf{c}=h_0=0$. Why this step? $\mathbf{c}=h_{T_x}^{enc}$ with $T_x=0$ means "the state before any token" — the initial state. It is not undefined; it is the neutral $h_0$. In practice the model would then just decode from a blank context (often producing $⟨\text{END}⟩$ immediately).

2. Single token, step 1: $h_1=\tanh (1\cdot 3+0.5\cdot 0)=\tanh (3)\approx 0.9951$. Why this step? One token means the recursion runs exactly once; memory term is zero because $h_0=0$.

3. Context: $\mathbf{c}=h_1\approx 0.9951$ — saturated near $+1$. Why this step? Large positive pre-activation ($3$) drives $\tanh$ into its flat region.

4. Answer: (a) $\mathbf{c}=0$; (b) $\mathbf{c}\approx 0.9951$.

Verify: $\tanh (0)=0$ ✓ and $\tanh (3)=0.99505\dots$ ✓, both inside $(-1,1)$. The single-token case saturates — a hint that large inputs get squashed, foreshadowing the bottleneck problem. ✓

---

Example 8 — Cell H: the information bottleneck limit

Forecast: Is the fixed vector short of capacity at 50 tokens, at 1000 tokens, or only far beyond that?

1. Bits needed per token. Using the log rule ${\log }_2(V^{T_x})=T_x\cdot {\log }_{2}V$, each of the $T_x$ positions independently chooses one of $V$ words, costing ${\log }_{2}V$ bits. So ${\log }_{2}V={\log }_2(10^4)=4{\log }_{2}10\approx 4\times 3.3219=13.288\text{ bits per token.}$ Why this step? Encoding one choice out of $V$ equally-possible options costs exactly ${\log }_{2}V$ bits — that is the definition of a bit as a "yes/no halving." The base-2 log is chosen (not natural log) precisely because we measure storage in bits, and one bit halves the number of remaining possibilities. Total sequence cost is then $T_x\times {\log }_{2}V$.

2. Bits available in the context vector. A float32 number stores $32$ bits; a $512$-dimensional vector stacks $512$ of them: $\text{bits available}=512\times 32=16,384\text{ bits.}$ Why this step? This is the entire budget the encoder has to describe the whole input — everything the decoder ever sees about $\mathbf{x}$ must fit in these $16,384$ bits. That hard ceiling is the bottleneck made literal.

3. Crossover length (where "needed" first exceeds "available"): solve $T_x\times 13.288=16,384$, giving $T_x=\frac{16,384}{13.288}\approx 1233\text{ tokens.}$ Why this step? Below $\approx 1233$ tokens the vector has enough raw bits to (in principle) store the sequence; above it, the vector is provably too small — no matter how clever the encoder, some sequences must collide onto the same $\mathbf{c}$, so information is guaranteed lost. This is the mathematical seed of the bottleneck.

4. Deficit at $T_x=2000$. Bits needed $=2000\times 13.288=26,576$; available $=16,384$; so $\text{deficit}=26,576-16,384=10,192\text{ bits.}$ Why this step? At 2000 tokens the vector falls short by over ten thousand bits — nearly 40% of the required information cannot physically be stored, so it is guaranteed lost. That lost information is exactly the "early tokens get forgotten" failure the parent described.

5. Limiting behaviour ($T_x\to \infty$). Bits needed grow linearly in $T_x$ ($13.288T_x$), while bits available stay constant at $16,384$. So the ratio (needed / available) $\to \infty$: the fraction of information the vector can preserve $\to 0$. Why this step? This is the crisp statement of the limit — a fixed-size $\mathbf{c}$ cannot keep pace with unboundedly long inputs, which is why attention lets the decoder read all encoder states rather than one squeezed vector.

6. Answer: (a) $16,384$ bits available; (b) crossover at $T_x\approx 1233$ tokens; (c) a $10,192$-bit deficit at $T_x=2000$.

Verify: $16,384/13.288\approx 1232.9$ ✓; and $2000\times 13.288=26,576$, minus $16,384$ gives $10,192$ ✓. The bottleneck is a genuine limiting-length phenomenon: short sentences fit comfortably, but past $\approx 1233$ tokens the fixed vector cannot cope, and as $T_x\to \infty$ its stored fraction of the input $\to 0$ — which is precisely why attention (3.5.09-Attention-mechanism) was invented. ✓

Recall Self-test

What is the total number of recurrent-cell applications for a 4-token input and a 6-token output (including END)? ::: Encoder 4 + decoder $6+1=7$, total $11$. Why does softmax use $e^z$ rather than $z$ directly? ::: $e^z>0$ always (no negative probabilities) and it exaggerates score gaps, sharpening the distribution. Perplexity of a model that assigns $0.1$ joint prob to a 3-token target? ::: $0.1^{-1/3}\approx 2.154$. Greedy picked "The dog" ($0.15$); which path beats it and by how much? ::: "A fox" at $0.20$, a margin of $0.05$. Crossover length where a 512-dim float32 context runs out of bits (V=10^4)? ::: About $1233$ tokens. As $T_x\to \infty$ with a fixed context vector, what happens to the stored fraction of the input? ::: It tends to $0$ — needed bits grow linearly, available bits stay constant.