Exercises — Sequence-to-sequence models

Count: exactly one context vector. In the basic architecture, $\mathbf{c}$ is the final encoder hidden state $h_{T_x}^{(enc)}$ — a single fixed vector no matter how long the input is. (Attention is what would give a different vector per step; we do not have it here.)

Dimension: $\mathbf{c}\in {\mathbb{R}}^{256}$. As defined above, the hidden size $d_h$ is the number of entries in every hidden state, and $\mathbf{c}=h_{T_x}^{(enc)}$ has that same length, so $d_h=256\Rightarrow \mathbf{c}\in {\mathbb{R}}^{256}$.

The mismatch $T_x=6\ne T_y=4$ is exactly the point of seq2seq: the fixed vector decouples the two lengths.

• $x_t$ ::: the $t$-th input token fed to the encoder.
• $\mathbf{c}$ ::: the context vector, the compressed "meaning" of the whole input.
• $y_{t-1}$ ::: the previous output token, fed back into the decoder to produce the next output $y_t$.
• $\text{<SOS>}$ ::: the start-of-sequence token that kicks off decoding (there is no $y_0$ otherwise).
• $\text{<EOS>}$ ::: the end-of-sequence token; when the decoder emits it, generation stops.

$\mathcal{L}=-(\ln 0.80+\ln 0.50+\ln 0.90)$ Step by step:

• $\ln 0.80=-0.22314$
• $\ln 0.50=-0.69315$
• $\ln 0.90=-0.10536$

Sum of logs $=-1.02165$, so $\mathcal{L}=+1.02165\approx 1.022$.

Why negative: each $\log P$ is negative (probabilities $\le 1$), so negating gives a positive loss. A perfect model with all $P(y_t^{\ast })=1$ (full confidence in every correct token) would give $\mathcal{L}=0$.

$\mathcal{L}=-\ln 0.92-\ln 0.93-\ln 0.94$

• $-\ln 0.92=0.08338$
• $-\ln 0.93=0.07257$
• $-\ln 0.94=0.06188$

Sum $=0.21783\approx 0.218$.

Where does the parent note's $\approx 0.24$ come from? It rounds each term to two decimals before adding: $-\ln 0.92\approx 0.08\to 0.08$, $-\ln 0.93\approx 0.07$, $-\ln 0.94\approx 0.06$, and then pads for a comfortable upper estimate — a quick "back-of-envelope" figure. Rounding early like this accumulates error, which is why the precise sum is the smaller $0.218$. Prefer the precise value ($0.218$); the parent's $0.24$ is only a rough sanity check, acceptable when you just want to confirm "the loss is small." Either way the message holds: confident correct predictions give a small loss.

Softmax turns logits into a probability distribution: $P(0)=\frac{e^{z_0}}{{\sum }_{v\in V}{e}^{z_v}}.$ Here the summation index $v$ runs over every token in the vocabulary $V$ (all 5 of them, indices $0$ through $4$); $z_v$ is the logit of token $v$. So the denominator adds up the exponential of each token's score. Exponentials: $e^2=7.389$, $e^1=2.718$, $e^0=1$, $e^{-1}=0.368$, $e^1=2.718$. Denominator $=7.389+2.718+1+0.368+2.718=14.193$. $P(0)=\frac{7.389}{14.193}=0.5206\approx 0.521.$ Why softmax: we must pick one token from $V$, so the outputs must be non-negative and sum to 1 — a probability distribution over the vocabulary.

(a) Greedy commits to the highest token at each step, ignoring the future. Step 1: "A" (0.6 > 0.4). Step 2 from "A": "y" (0.7 > 0.3). Greedy output: "A y", probability $0.6\times 0.7=0.42$.

(b) Beam search, width $k=2$ — follow the actual pruning, step by step:

Step 1 — score the length-1 hypotheses and keep top $k=2$. Candidates: "A" $=0.6$, "B" $=0.4$. There are only two, so both survive. Beam after step 1: {"A" (0.6), "B" (0.4)}.

Step 2 — expand every surviving hypothesis, then prune back to $k=2$. Expand "A": "A x" $=0.6\times 0.3=0.18$, "A y" $=0.6\times 0.7=0.42$. Expand "B": "B z" $=0.4\times 0.9=0.36$, "B w" $=0.4\times 0.1=0.04$. Now we have four candidates {0.42, 0.36, 0.18, 0.04}; prune to the top 2 → keep "A y" (0.42) and "B z" (0.36). The candidates "A x" and "B w" are discarded here — this pruning is the whole point of beam search (it never keeps more than $k$ live paths).

Both survivors are complete length-2 sequences, so beam's best output is "A y" (0.42).

What pruning bought us: notice "B z" (0.36) rose to second place even though its first token "B" was weaker — it survived because we carried more than one hypothesis. Greedy would have thrown "B" away at step 1.

When greedy fails: set $P(\text{"y"}\mid \text{A})=0.4$ instead. Greedy still picks "A" then its best child (0.4), giving "A y" $=0.6\times 0.4=0.24$. But beam keeps "B" alive and finds "B z" $=0.4\times 0.9=0.36>0.24$. Beam wins, greedy misses it.

(c) Greedy is locally optimal but can be globally wrong: a strong first token can lead only to weak continuations. Beam search hedges by carrying — and pruning down to — several hypotheses per step.

Raw log-prob: $P$ has $-0.9>-1.4$, so P wins (less negative = higher probability).

Length-normalized:

• $P$: $\frac{-0.9}{2}=-0.45$
• $Q$: $\frac{-1.4}{4}=-0.35$

Now $Q$ wins ($-0.35>-0.45$).

Why the flip: every extra token multiplies in another probability $<1$, so raw score always favours shorter outputs — the model would learn to stop early. Dividing by length $t$ removes this bias, letting a longer-but-per-token-confident sequence win.

The setup: during training we feed the ground-truth previous token $y_{t-1}^{\ast }$ at every step. The decoder therefore never practices recovering from its own errors — this is exposure bias.

The cascade (inference): suppose target is "Estoy muy feliz".

• $t=1$: model predicts "Estoy" ✓.
• $t=2$: model predicts "contento" instead of "muy" (a plausible slip).
• $t=3$: now the decoder's input is its own "contento", a context it saw rarely in training. It predicts "$\text{<EOS>}$" and stops. Output: "Estoy contento" — grammatical but wrong, and the error at $t=2$ propagated.

Why training loss looked fine: at training the decoder always received the correct "muy", so it never entered the off-track state and never paid for it.

Mitigation: scheduled sampling — during training, with growing probability feed the model's own prediction instead of ground truth, so it learns to recover. (Beam search at inference also helps by not committing to one early slip.)

(a) Concatenating forward and backward states of size 128 each gives $128+128=256$ per step. If you form $\mathbf{c}$ by concatenating the two final states, $\mathbf{c}\in {\mathbb{R}}^{256}$.

(b) A forward-only encoder at token $t$ has seen only $x_1,\dots ,x_t$ — the past. But meaning often depends on the future. In "the bank of the river", the word "bank" is ambiguous until you read "river". The backward LSTM $\overleftarrow{h}$ reads right-to-left, so its representation of "bank" already contains "river". Concatenating both directions gives each token a representation informed by the entire sentence, not just its left context. See LSTM networks.

(a) Demand vs capacity.

• Capacity of $\mathbf{c}$: $512\text{ dims}\times 4\text{ bits}=2048$ bits — this is fixed, independent of input length.
• Demand, 50 tokens: $50\times 10=500$ bits. Since $500<2048$, the vector has room to spare — the bottleneck is not yet saturated. This matches the parent note's remark that Sutskever et al. (2014) got good results on moderate-length sentences.
• Demand, 400-token document: $400\times 10=4000$ bits. Now $4000>2048$: the vector physically cannot hold everything, so some information (typically the earliest tokens, which the LSTM tends to forget) must be overwritten.

(b) What it predicts: translation/summarisation quality with a fixed $\mathbf{c}$ should be fine for short inputs but degrade as inputs grow long — exactly the empirical curve reported for basic seq2seq.

How attention escapes the bound: instead of squeezing everything through one fixed vector, the decoder computes a fresh context vector ${\mathbf{c}}_t$ at each output step. Concretely, it takes a weighted sum of all encoder hidden states $h_1^{(enc)},\dots ,h_{T_x}^{(enc)}$, with weights ("attention scores") that say how relevant each input position is right now. Two consequences:

• The information the decoder can draw on is no longer a single $d_h$-vector but the whole set $T_x\times d_h$ — capacity now grows with input length ($400\times 512\times 4=819,200$ bits available, far exceeding the $4000$ demanded).
• There is no forced forgetting of early tokens: if step $t$ needs $x_1$, its attention weight on $h_1^{(enc)}$ simply rises.

So attention removes the fixed-size bottleneck entirely, which is why it is the natural sequel to this topic (and the foundation of transformers).

Trace the path (backwards). The loss at the last decoder step $t=T_y$ depends on $h_{T_y}^{(dec)}$, which depends on $h_{T_y-1}^{(dec)}$, ..., back to $h_1^{(dec)}$ — that is $\approx T_y$ recurrent steps inside the decoder. The first decoder state depends on the context vector $\mathbf{c}=h_{T_x}^{(enc)}$. From there the gradient chains back through the encoder: $h_{T_x}^{(enc)}\to h_{T_x-1}^{(enc)}\to \cdots \to h_1^{(enc)}\to x_1$ — that is $\approx T_x$ more recurrent steps.

The count. Total recurrent multiplications $\approx T_y+T_x$ — the gradient must survive the full decoder plus the full encoder, chained end to end through two RNNs.

Why this is the danger zone. At each recurrent step, backprop multiplies the incoming gradient by that step's recurrent Jacobian (the derivative of one hidden state with respect to the previous one). Chaining $T_x+T_y$ such factors means the gradient reaching $x_1$ is a product of $T_x+T_y$ matrices. If those factors have magnitude consistently less than 1, the product shrinks geometrically toward zero — the gradient vanishes before it reaches the early encoder inputs, so the model barely learns to use $x_1$. (Conversely, factors $>1$ make it explode.) This is precisely why plain RNN cells are replaced by LSTM/GRU gates, whose near-identity gradient paths keep the product near 1 — and why very long sequences ultimately favour transformers, whose attention gives a constant-length (length-1) gradient path between any two positions.

Basic seq2seq passes the decoder how many context vectors? ::: Exactly one — the final encoder state. Which log base does ML cross-entropy use? ::: Natural log ($\ln$). Why can't the decoder be bidirectional? ::: Future output tokens don't exist yet at generation time.