Exercises — Encoder-decoder architecture

• $x_t$ ::: the input token (as a vector) fed upward into a teal encoder box at step $t$.
• $h_t^{enc}$ ::: the encoder's hidden state — a teal box — a memory summarising input tokens $1$ through $t$.
• $\mathbf{c}$ ::: the context vector, the plum box in the middle: the single fixed-size summary the encoder hands to the decoder. Usually $\mathbf{c}=h_{T_x}^{enc}$ (in the figure, $h_3^{enc}$).
• $h_t^{dec}$ ::: the decoder's hidden state — an orange box — memory of what it has generated so far.
• $y_t$ ::: the output token emitted upward from an orange box.
• $T_x$ ::: number of input tokens, i.e. the number of teal boxes ($3$ in the figure).
• $T_y$ ::: number of output tokens, i.e. the number of orange boxes. Crucially $T_x$ and $T_y$ can differ.

False. The entire point of this architecture is decoupling: 5 input words can map to 7 output words — i.e. 5 teal boxes feeding 7 orange boxes. The context vector $\mathbf{c}$ (plum box) is the fixed-size bridge that lets $T_x\ne T_y$.

(b) $\mathbf{c}=h_{T_x}^{enc}$ — the final teal box (the plum arrow in the figure leaves $h_3^{enc}$), because by the last step it has recursively "seen" every input token.

Softmax turns scores into probabilities via $P(v)=\frac{e^{z_v}}{{\sum }_u{e}^{z_u}}$. Why exponentiate? $e^z$ is always positive and monotonically increasing, so it maps any real score to a positive weight while preserving the ranking; dividing by the sum then normalises the weights to add to $1$. Numerators: $e^{2.0}=7.389$, $e^{1.0}=2.718$, $e^{0.0}=1.000$. Sum $=11.107$.

• $P(\text{"am"})=7.389/11.107=0.6652$
• $P(\text{"is"})=2.718/11.107=0.2447$
• $P(\text{"the"})=1.000/11.107=0.0900$ They sum to $1$ (a valid distribution). Greedy picks the argmax, i.e. "am" (highest probability $0.665$).

Why a product? The decoder is autoregressive: each output token depends on all previous ones, so the joint probability of the whole sequence factorises by the chain rule as $P(y_1,y_2,y_3\mid \mathbf{x})=P(y_1\mid \mathbf{x})P(y_2\mid y_1,\mathbf{x})P(y_3\mid y_1,y_2,\mathbf{x}).$ Each listed probability is exactly one conditional factor. Multiplying: $P({\mathbf{y}}^{\ast }\mid \mathbf{x})=0.5\times 0.4\times 0.25=\mathrm{0.05.}$ Taking the natural log turns product into a sum (why we log: avoids tiny-number underflow, see L5.2): $\log P=\ln 0.5+\ln 0.4+\ln 0.25=(-0.6931)+(-0.9163)+(-1.3863)=-\mathrm{2.9957.}$

$\mathcal{L}=-\log P({\mathbf{y}}^{\ast }\mid \mathbf{x})=-(-2.9957)=\mathrm{2.9957.}$ Why this is cross-entropy. At each step the target distribution is a one-hot vector (probability $1$ on the true token $y_t^{\ast }$, $0$ elsewhere). Cross-entropy between target $q$ and model $p$ is $-{\sum }_{v}q(v)\log p(v)$; with one-hot $q$ this collapses to just $-\log p(y_t^{\ast })$ — the negative log-probability of the correct token. Summing over $t$ gives exactly $\mathcal{L}$ above. So "negative log-likelihood" and "cross-entropy against one-hot targets" are the same number. Lower loss ⇔ higher probability assigned to the correct sequence. The training objective $\mathcal{L}(\theta )=-\frac{1}{N}{\sum }_i{\sum }_t\log P_{\theta }(\cdot )$ is exactly this, averaged over tokens and examples.

(a) Multiply along each path:

• Path A: $0.5\times 0.3=0.15$.
• Path B: $0.4\times 0.5=0.20$.

(b) Greedy compares only step-1 scores: $0.5>0.4$, so it commits to "The" → locked onto Path A. (c) Globally, Path B ($0.20$) beats Path A ($0.15$). (d) Locally optimal ≠ globally optimal. A high-probability first token can lead to low-probability continuations. This motivates beam search, which keeps several hypotheses alive instead of committing early.

Step 1: rank candidates → keep top 2: "Fox" ($0.3$) and "The" ($0.2$). "A" ($0.15$) is pruned. Step 2: expand each surviving beam, multiplying probabilities:

• "Fox jumps" $=0.3\times 0.4=0.12$
• "Fox runs" $=0.3\times 0.1=0.03$
• "The fox" $=0.2\times 0.3=0.06$
• "The dog" $=0.2\times 0.25=0.05$

Keep top 2 overall: "Fox jumps" ($0.12$) and "The fox" ($0.06$). Note greedy would also have started with "Fox" but beam keeps "The fox" as a live backup in case its later continuations dominate.

(a) — step by step.

• There are $V$ choices for each of $T_x$ positions, so the number of distinct sequences is $V^{T_x}=(10^4{)}^{50}=10^{200}$.
• To give each sequence a unique binary address ("index"), you need ${\log }_2(\text{number of sequences})$ bits. Why ${\log }_2$? $b$ bits label exactly $2^b$ distinct things, so to label $N$ things you invert this: $b={\log }_{2}N$.
• Compute: ${\log }_2(V^{T_x})=T_x{\log }_{2}V=50\times {\log }_2(10^4)$. Now ${\log }_2(10^4)=4{\log }_{2}10=4\times 3.3219=13.2877$. So $b=50\times 13.2877=664.39$ bits.

So merely to index which of the $10^{200}$ sequences you mean already costs about 664 bits — and that assumes a perfect, lossless code. To reconstruct the meaning robustly (not just index it) a real encoder needs comfortably more headroom than this bare minimum.

(b) A float32 number is 32 bits, and there are 512 of them: $512\times 32=16,384$ bits of raw storage.

(c) Here $16,384$ bits is far more than the $664$-bit index, so for a single length-50 sequence the vector has room. The bottleneck bites when sequences get long and you demand robust, lossy-tolerant meaning rather than a perfect index: the required information grows linearly in $T_x$ ($664$ bits at $T_x=50$, doubling by $T_x=100$, and so on), while the context vector stays a fixed $16,384$ bits forever. Past some length the fixed budget is overwhelmed — the classic information bottleneck. This is what attention fixes by letting the decoder read all encoder states instead of one squeezed summary.

(a) First evaluate the embedding: $\text{emb}(y_0)=0.1$ is the vector (here scalar) that represents the START token $y_0$. Pre-activation: $W_h{h}_0^{dec}+W_{hy}\text{emb}(y_0)+b_h=1.0(0.5)+2.0(0.1)+0=\mathrm{0.7.}$ Apply $\tanh$ (squashes any real number into $(-1,1)$ — why: keeps hidden states bounded and centred so they neither explode nor vanish across steps): $h_1^{dec}=\tanh (0.7)=\mathrm{0.6044.}$ (b) Feed this into the output layer to get the logit: $z=W_o{h}_1^{dec}+b_o=3.0\times 0.6044+(-0.5)=\mathrm{1.3131.}$ In a real network $z$ would be a whole vector of such scores (one per vocabulary word) fed into softmax; here we computed the single scalar version to see the mechanics.

(a) Teacher forcing feeds the ground-truth previous token → feed "love" (the model's mistake is ignored for the input, though still penalised in the loss). (b) Free running feeds the model's own output → feed "adore", pushing the decoder into a state it may never have trained on. (c) Test time is always free-running. A model trained only with teacher forcing never practised recovering from its own errors → exposure bias. Fixes: scheduled sampling (mix in own predictions during training) and professor forcing (match teacher-forced and free-running hidden-state distributions).

No. L3.3 showed the required index information grows linearly with length while the context vector stays a fixed $16,384$ bits; performance degrades sharply past ~20 words. The fix is the attention mechanism (3.5.09-Attention-mechanism): the decoder computes a fresh weighted blend of all encoder states $h_1^{enc},\dots ,h_{T_x}^{enc}$ at each step, so early tokens are no longer squeezed through one bottleneck. This idea generalises into the Transformer.

(a) Yes. With $T_x=1$, $\mathbf{c}=h_1^{enc}=f_{enc}(x_1,h_0^{enc})$ — the recursion runs exactly once (one teal box). Perfectly valid; no degeneracy. (b) Ruled out. With $T_x=0$ the encoder runs zero steps, so there is no $h_{T_x}^{enc}$ to become $\mathbf{c}$; the only state available is the initial $h_0^{enc}$ (all zeros), which carries no information. In practice this case is never fed — inputs must contain at least one real token (often a START marker is prepended, giving $T_x\ge 1$). So an empty-input translation is undefined and excluded by construction. (c) An "empty output" is not truly empty: the decoder emits $⟨\text{END}⟩$ immediately at step 1. So $T_y=1$ in practice (the END marker). A genuinely zero-length sequence is undefined for an autoregressive decoder — there must be at least the stop signal. (d) $⟨\text{END}⟩$ is the learned stop signal — sampling it halts generation. If the decoder never assigns high probability to END, generation runs until a hard max-length cap forces a stop, often producing rambling or repeated output. This is why max-length limits are mandatory in practice.

(a) The joint probability is the product of the 20 factors: $0.5^{20}=9.5367\times 10^{-7}$. (Still representable here — but watch what happens as length grows.) (b) Log turns the product into a sum: $\ln (0.5^{20})=20\ln 0.5=20\times (-0.6931)=-13.8629$. (c) At length 20 the raw value $9.5\times 10^{-7}$ is fine. But the raw product decays exponentially in length: at length $\approx 130$ we'd hit $0.5^{130}\approx 7.3\times 10^{-40}$, which is below float32's smallest positive normal $1.2\times 10^{-38}$ → it underflows to exactly $0$, and then $\ln 0=-\infty$ destroys the gradient. The log, by contrast, decays only linearly: at length 130 it is a comfortable $\ln (0.5^{130})=130\times (-0.6931)\approx -90.1$, nowhere near any limit. This justifies why the loss sums logs instead of multiplying probabilities — numerical stability under long sequences.

Let $q$ be the projection that ignores all but the last state: $q(\{h_1,\dots ,h_{T_x}\})=h_{T_x}$. Then $\mathbf{c}=q(\cdot )=h_{T_x}^{enc}$ — the special case. The general $q$ can be any pooling: mean-pool $\frac{1}{T_x}{\sum }_t{h}_t^{enc}$, max-pool, or — the powerful version — a content-dependent weighted sum ${\sum }_t{\alpha }_t{h}_t^{enc}$ where weights ${\alpha }_t$ change per decoder step. That last form is attention: it lets $\mathbf{c}$ adapt to what the decoder currently needs, dissolving the single-vector bottleneck.

Why sum logs instead of multiplying probabilities in the loss? ::: Products of many sub-1 probabilities underflow to zero in floating point; logs turn the product into a sum that decays only linearly and stays representable. What single change removes the fixed-context bottleneck? ::: Attention — a per-step weighted sum ${\sum }_t{\alpha }_t{h}_t^{enc}$ over all encoder states instead of one final vector. What causes exposure bias? ::: Training only with teacher forcing, so the decoder never practises recovering from its own mistakes it will face at test time. Why subtract the max logit before softmax? ::: To stop large exponentials overflowing to infinity; it is algebraically identical since the shared factor cancels.