Sequence-to-sequence models: Worked examples — AI-ML deep dive

Before anything else, some symbols we will reuse. If you have never seen them, read this box slowly — every later example depends on it.

Definition The symbols we lean on

$P (y_{t} ∣ \dots)$ means "the probability the model assigns to token $y_{t}$ ". A probability is just a number between $0$ and $1$ ; all the numbers over the whole vocabulary add up to $1$ . Think of it as slicing a pie: each token gets a slice, the slices fill the pie.
$y_{< t}$ is shorthand for all the tokens that came before step $t$ — that is, $y_{1}, y_{2}, \dots, y_{t - 1}$ , the output produced so far. So " $P (y_{t} ∣ y_{< t})$ " reads "the probability of the next token given everything already emitted".
$T_{x}$ is the length of the input sequence (number of input tokens) and $T_{y}$ is the length of the target/output sequence. The subscript $x$ points at the input, $y$ at the output.
$lo g$ is the logarithm (natural log, base $e \approx 2.718$ ). It answers "what power do I raise $e$ to, to get this number?" We use it because $lo g$ turns a product of probabilities into a sum — and computers add more safely than they multiply tiny numbers. Also $lo g$ of a number below $1$ is negative, and the closer the probability is to $1$ , the closer $lo g$ is to $0$ .
$ar g max_{v}$ means "the token $v$ that makes the thing biggest" — not the biggest value, but the identity of the winner.
$k$ is the beam width — the number of candidate partial sequences a beam search keeps alive at each step. $k = 1$ collapses back to greedy; larger $k$ explores more paths. We use it in Example D.

The scenario matrix

Every worked example below is tagged with the cell it covers.

#	Cell (situation)	What could go wrong / why it's distinct
A	Short input, greedy decode	Baseline — one token per step, always pick the max
B	Cross-entropy loss numbers	Turning three probabilities into one loss number
C	Confident vs confused model	Same math, different confidence → very different loss
D	Beam search beats greedy	Greedy's locally-best first word leads to a dead end
E	Length normalization	Without it, beam search prefers short garbage
F	Degenerate: `<EOS>` on step 1	The empty-output edge case
G	Teacher forcing vs free-running	Same step, two different inputs → exposure bias
H	Long input / bottleneck limit	Why a fixed $c$ eventually fails (limiting behaviour)
I	Real-world word problem	End-to-end translation, all pieces at once
J	Exam twist: pad/mask a batch	Variable lengths in one batch

Example A — short input, greedy decode

Forecast: guess the output sentence before reading on.

Step 1: take $ar g max$ of $P_{1}$ . Why this step? Greedy decoding means "commit to the single most probable token now" — $ar g max_{v} P (v)$ . The biggest entry of $P_{1}$ is $0.92$ at index $1 = Estoy$ .
Step 2: $ar g max$ of $P_{2}$ . Why? Same rule at the next time step; the previous winner \texttt{Estoy} was fed back in. Biggest is $0.93$ at index $2 = feliz$ .
Step 3: $ar g max$ of $P_{3}$ . Why? Biggest is $0.94$ at index $3 = <EOS>$ — this is the stop signal, so we halt.

Figure below: three bar charts, one per decoding step. Each bar is a token's probability; the tallest bar — the one greedy commits to — is drawn in red, with an "argmax" arrow. It shows visually that greedy simply keeps grabbing the red bar and never looks back.

Output: \texttt{Estoy feliz } → "Estoy feliz".

Verify: three probabilities are all $> 0.9$ , so greedy and any sensible strategy agree here — no risk. Indices $1, 2, 3$ decoded in order. ✓

Example B — cross-entropy loss from those numbers

Forecast: will the loss be near $0$ or near $1$ ? (High confidence → low loss.)

Pick out the probability of the correct token at each step. Why? The loss only cares how much mass the model put on the right answer: $0.92, 0.93, 0.94$ .
Take $- lo g$ of each. Why? $- lo g$ converts "probability of correct" into "penalty": a probability of $1$ gives penalty $0$ ; smaller probabilities give larger penalties. $- lo g 0.92 \approx 0.0834, - lo g 0.93 \approx 0.0726, - lo g 0.94 \approx 0.0619$
Sum them. Why? The whole-sequence probability is a product, and $lo g$ of a product is a sum, so total loss adds the per-step penalties. $L \approx 0.0834 + 0.0726 + 0.0619 = 0.2179 \approx 0.22$

Verify: all three probabilities are close to $1$ , so loss should be small — and $0.22$ is indeed tiny (a random 4-way guess would give $- 3 lo g (0.25) \approx 4.16$ ). ✓

Example C — same math, a confused model

Forecast: the loss should be much larger — the model is barely favouring the right words.

$- lo g$ each correct probability. Why? Same penalty rule. $- lo g 0.40 \approx 0.916, - lo g 0.35 \approx 1.050, - lo g 0.50 \approx 0.693$
Sum. $L \approx 0.916 + 1.050 + 0.693 = 2.659 \approx 2.66$ .

Figure below: the black curve is the penalty $- lo g p$ plotted against the probability $p$ of the correct token. The confident case (probabilities near $0.93$ ) sits low on the curve; the confused case (near $0.42$ , marked in red) sits high where the curve rears up. It shows why a small drop in confidence causes a large jump in loss: the penalty curve is steep near small $p$ .

Verify: $2.66 ≫ 0.22$ . Same three-step structure, but confidence dropped from $\sim 0.93$ to $\sim 0.42$ , so loss climbed roughly $12 \times$ . The red points on the steep part of the curve explain why. ✓

Example D — beam search beats greedy

Forecast: greedy picks \texttt{Estoy} first; will beam agree?

Greedy step 1. Why? Greedy commits to $ar g max = Estoy$ ( $0.5 > 0.45$ ) and can never reconsider.
Greedy full score = $0.5 \times 0.30 = 0.15$ . Why multiply? Sequence probability is the product of step probabilities.
Beam keeps both \texttt{Estoy} and \texttt{Yo} after step 1. Why? Beam width $k = 2$ retains the top- $2$ partial paths instead of one.
Score each 2-token path. Why compute these scores? Beam ranks whole partial sequences by their probability so far, not just the newest token — this is the only way it can discover that a slightly-worse first word (\texttt{Yo}) leads to a far better second word. We multiply each first-token probability by its best continuation:
- \texttt{Estoy …}: $0.5 \times 0.30 = 0.15$
- \texttt{Yo soy}: $0.45 \times 0.95 = 0.4275$
Pick the max path. Why? Beam returns the highest-scoring complete sequence: \texttt{Yo soy} at $0.4275$ .

Verify: $0.4275 > 0.15$ , so beam finds a sequence nearly $3 \times$ more probable than greedy's — exactly the payoff of keeping options open. See 4.2.01-Beam-search. ✓

Example E — length normalization

Forecast: raw scoring will unfairly reward the shorter sentence.

Raw totals. Why? Adding log-probs = multiplying probabilities. $A : - 0.10 - 0.20 = - 0.30$ . $B : - 0.10 - 0.30 - 0.20 = - 0.60$ .
Raw ranking: $A (- 0.30) > B (- 0.60)$ , so raw prefers the shorter $A$ . Why is that bad? Every extra token adds a negative number, so longer is punished automatically — even good long translations.
Divide by length $t$ . Why? Averaging removes the "more tokens = more penalty" bias. $A : - 0.30/2 = - 0.15$ . $B : - 0.60/3 = - 0.20$ .
Normalized ranking: $A (- 0.15) > B (- 0.20)$ — here $A$ still wins, but the gap shrank from $0.30$ to $0.05$ , showing the correction at work.

Verify: normalization brought the scores closer ( $0.30 \to 0.05$ gap), confirming it counteracts the length bias without flipping a genuinely-better short answer. ✓

Example F — degenerate: `<EOS>` on the very first step

Forecast: what happens if the model wants to stop immediately?

$ar g max P_{1}$ . Why? Greedy rule. Max is $0.80$ at \texttt{}.
We halt at step 1. Why? <EOS> is the stop signal, so generation ends before any real word is emitted.
Output = empty string (just \texttt{}). Why care? This is the degenerate/zero-length case — real systems clamp a minimum length so an over-eager model can't return nothing.

Verify: the single decoded index is $3 = <EOS>$ , length of real output $= 0$ . A minimum-length guard would forbid index $3$ on step 1 and force the runner-up (\texttt{Hola}, $0.10$ ). ✓

Example G — teacher forcing vs free-running (same step, two inputs)

Forecast: which input does each mode use?

Training / teacher forcing: feed the ground-truth \texttt{Estoy}. Why? Correct context each step stops early errors from snowballing and speeds convergence. See BPTT.
Inference / free-running: feed the model's own \texttt{Yo}. Why? At test time there is no ground truth — the model must consume its own output.
The mismatch = exposure bias. Why it matters? The model trained only on correct history, so at inference a wrong \texttt{Yo} pushes it into states it never practised.

Verify: the two inputs differ (\texttt{Estoy} vs \texttt{Yo}) → non-empty discrepancy → this is exactly the exposure-bias scenario the parent note warned about. ✓ (No number to compute; this cell is conceptual.)

Example H — long input, the bottleneck limit

Forecast: as $T_{x}$ grows, does a fixed $c$ keep up?

Total input capacity. Why? Each token carries $256$ numbers; $T_{x} = 50$ tokens carry $50 \times 256 = 12800$ numbers.
Context capacity is fixed at $512$ . Why fixed? By construction $c = h_{T_{x}}^{(e n c)}$ has one size regardless of $T_{x}$ .
Compression ratio $= 12800/512 = 25$ . Why care? The encoder must squeeze $25 \times$ its own size into $c$ . As $T_{x} \to \infty$ the ratio $\to \infty$ , so early tokens must eventually be forgotten — the information bottleneck.

Figure below: two bars — a tall black bar for the $12800$ numbers the input carries, and a short red bar for the $512$ slots inside $c$ — with a "bottleneck" arrow pointing at the small red one. It makes the $25 \times$ mismatch visible at a glance and shows why long inputs overflow a fixed context vector.

Verify: ratio $= 25$ ; doubling $T_{x}$ to $100$ doubles it to $50$ while $c$ stays $512$ — the limit that motivates the attention mechanism and later Transformers. ✓

Example I — real-world word problem (end to end)

Forecast: which hidden state becomes $c$ , and what's the final loss?

(a) Context vector = last encoder state. Why the last? It is the only state that has "seen" all $T_{x} = 3$ tokens: $c = h_{3} = [0.1, 0.6, - 0.3, 0.4]$ . (The values are arbitrary placeholders chosen to make the shape concrete — only their count, $d_{h} = 4$ , matters here.)
(b) Decode greedily as in Example A. Each step takes $ar g max$ of its distribution and feeds the winner forward, giving \texttt{Estoy}, then \texttt{feliz}, then \texttt{} → "Estoy feliz". Why? Same greedy rule; the final \texttt{} halts generation.
(c) Loss as in Example B: $L = - lo g 0.92 - lo g 0.93 - lo g 0.94 \approx 0.2179 \approx 0.22$ . Why? Sum the per-step $- lo g$ penalties of the correct tokens.

Verify: $c$ has $d_{h} = 4$ entries ✓; output is a valid Spanish rendering ending in <EOS> ✓; loss $\approx 0.22$ matches Example B ✓.

Example J — exam twist: pad and mask a batch

Forecast: should the padded positions add to the loss?

Pad the short sequence to length $4$ . Why? Every row in a batch tensor must share a shape; padding makes both sequences length $4$ so they stack into one array.
Build a mask = $[1, 1, 0, 0]$ for sequence 1 and $[1, 1, 1, 1]$ for sequence 2. Why? A $0$ marks a padded slot whose loss is meaningless and must be excluded from training.
Masked total loss = sum of losses only where the mask is $1$ . Why? Padded positions were never real target tokens, so counting them would teach the model to "predict" padding. Sequence 1: $0.1 + 0.2 = 0.3$ . Sequence 2: $0.1 + 0.3 + 0.2 + 0.2 = 0.8$ . Batch total $= 0.3 + 0.8 = 1.1$ .
Mean over real tokens. Why divide by real-token count, not $8$ ? Averaging over the $2 + 4 = 6$ genuine tokens keeps the loss scale comparable across batches with different padding. Mean $= 1.1/6 \approx 0.1833$ .
Contrast the buggy sum. Why mention it? Even though the padded losses happened to be $0.0$ here, a real model would emit nonzero garbage there; the mask guarantees $0$ contribution regardless, which is the whole point.

Verify: masked total $= 1.1$ ; counted positions $= 6$ ; mean $= 1.1/6 \approx 0.1833$ — averaging over real tokens only. ✓

Recall Quick self-test

Greedy decoding rule in one symbol ::: $y_{t} = ar g max_{v} P (v ∣ y_{< t}, c)$ Meaning of $y_{< t}$ ::: all output tokens before step $t$ , i.e. $y_{1}, \dots, y_{t - 1}$ What does the beam width $k$ control ::: how many partial candidate sequences beam search keeps alive at each step Why divide beam scores by length $t$ ::: to remove the bias against longer sequences (each token adds a negative log-prob) Loss for correct-probs $0.92, 0.93, 0.94$ ::: $\approx 0.22$ What input does teacher forcing feed at step $t$ ::: the ground-truth token $y_{t - 1}^{*}$ Compression ratio for $50 \times 256$ input into $c \in R^{512}$ ::: $25$ Masked batch total for losses $[0.1, 0.2]$ and $[0.1, 0.3, 0.2, 0.2]$ ::: $1.1$

Related: 3.5.01-Recurrent-neural-networks, 3.5.05-LSTM-networks, 3.5.06-GRU-networks, 3.5.07 Sequence-to-sequence models (Hinglish).