Level 4 — ApplicationSequence Models

Sequence Models

⏱ 60 minutes60 marksprintable — key stays hidden on paper

Level 4 (Application: Novel Problems, No Hints)

Time limit: 60 minutes
Total marks: 60

Answer all questions. Show all working. Calculators permitted for arithmetic. Give numeric answers to 4 decimal places where appropriate.

Question 1 — RNN forward pass & BPTT gradient (14 marks)

A vanilla RNN uses $h_t = \tanh(W_h h_{t-1} + W_x x_t)$ (no bias). You are given scalars (single-unit RNN): $W_h = 0.5$ , $W_x = 1.0$ , $h_0 = 0$ . The input sequence is $x_1 = 1.0$ , $x_2 = -0.5$ . The scalar loss is $L = \tfrac{1}{2}(h_2 - y)^2$ with target $y = 0$ .

(a) Compute $h_1$ and $h_2$ to 4 decimal places. (4)

(b) Using backpropagation through time, derive symbolically $\dfrac{\partial L}{\partial W_h}$ in terms of the computed activations, then evaluate it numerically. (7)

(c) State one reason why, for a 100-step version of this same RNN with $|W_h|=0.5$ , the gradient contribution from step 1 to the loss at step 100 would be negligible. (3)

Question 2 — LSTM cell computation (14 marks)

Consider a single-unit LSTM at one time step with the standard equations: $f_t=\sigma(z_f),\ i_t=\sigma(z_i),\ o_t=\sigma(z_o),\ \tilde{c}_t=\tanh(z_c)$ $c_t=f_t\odot c_{t-1}+i_t\odot \tilde{c}_t,\quad h_t=o_t\odot\tanh(c_t)$

You are given the pre-activation values $z_f = 2.0$ , $z_i = 0.0$ , $z_o = 1.0$ , $z_c = -1.0$ , and previous cell state $c_{t-1}=0.8$ .

(a) Compute $f_t, i_t, o_t, \tilde{c}_t$ (4 dp). (4)

(b) Compute $c_t$ and $h_t$ (4 dp). (4)

(c) Suppose during training the forget gate saturates so $f_t \approx 1$ and $i_t \approx 0$ across many steps. Explain what happens to the cell state and why this mitigates the vanishing-gradient problem that afflicts vanilla RNNs. (6)

Question 3 — Attention weights (Luong dot-product) (12 marks)

An encoder produces 3 hidden states $\bar{h}_1=[1,0]$ , $\bar{h}_2=[0,1]$ , $\bar{h}_3=[1,1]$ . The decoder state at the current step is $s=[1,1]$ . Luong (dot) attention scores each encoder state via $\text{score}=s\cdot\bar{h}_j$ .

(a) Compute the three raw scores. (3)

(b) Compute the softmax attention weights $\alpha_j$ (4 dp). (5)

Question 4 — Beam search decoding (10 marks)

A decoder has vocabulary $\{A, B, \text{<eos>}\}$ . Using beam width $k=2$ , the log-probabilities of the next token given the prefix are:

Step 1 (from start): $\log p(A)=-0.5,\ \log p(B)=-1.0,\ \log p(\text{<eos>})=-3.0$ .

Step 2 log-probs given the step-1 token:

given "A": $\log p(A)=-1.2,\ \log p(B)=-0.7,\ \log p(\text{<eos>})=-2.0$
given "B": $\log p(A)=-0.4,\ \log p(B)=-1.5,\ \log p(\text{<eos>})=-1.0$

(a) After step 1, list the 2 beams kept and their cumulative scores. (3)

(b) Expand both beams and list all candidate sequences of length 2 with cumulative log-probs. (4)

Question 5 — Design / conceptual application (10 marks)

You are building a machine-translation system (English→French) with an encoder–decoder RNN.

(a) Your training uses teacher forcing but inference does not. Describe the resulting exposure bias problem and one concrete mitigation. (4)

(b) Input sentences vary from 3 to 40 tokens. Explain how you batch them efficiently and why masking is required in the loss. (3)

Answer keyMark scheme & solutions

Question 1 (14 marks)

(a) $h_1 = \tanh(0.5\cdot 0 + 1.0\cdot 1.0) = \tanh(1.0) = 0.7616$ — (2) $h_2 = \tanh(0.5\cdot 0.7616 + 1.0\cdot(-0.5)) = \tanh(0.3808 - 0.5) = \tanh(-0.1192) = -0.1186$ — (2)

(b) Loss $L=\tfrac12(h_2-y)^2$ , $y=0 \Rightarrow \dfrac{\partial L}{\partial h_2}=h_2 = -0.1186$ . (1)

Chain through both time steps (BPTT). Let $\tanh'(a)=1-\tanh^2(a)$ . $\frac{\partial L}{\partial W_h}= \frac{\partial L}{\partial h_2}\cdot\left[\frac{\partial h_2}{\partial W_h}\Big|_{\text{direct}} + \frac{\partial h_2}{\partial h_1}\frac{\partial h_1}{\partial W_h}\right]$

$\frac{\partial h_2}{\partial W_h}|_{direct}=(1-h_2^2)\,h_1$ (the $W_h h_1$ term). (2)
$\frac{\partial h_2}{\partial h_1}=(1-h_2^2)\,W_h$ . (1)
$\frac{\partial h_1}{\partial W_h}=(1-h_1^2)\,h_0 = 0$ (since $h_0=0$ ). (1)

So the second path vanishes. Numerically: $1-h_2^2 = 1-0.01407 = 0.98593$ . $\frac{\partial L}{\partial W_h} = (-0.1186)(0.98593)(0.7616) = -0.0891$ . (2)

(c) The gradient propagated back $n$ steps scales like $\prod (1-h_t^2)\,W_h$ ; each factor $|W_h|=0.5<1$ and $(1-h^2)\le 1$ , so over 100 steps the product $\le 0.5^{100}\approx 10^{-30}$ — exponentially small ⇒ vanishing gradient, negligible contribution. (3)

Question 2 (14 marks)

(a) $f_t=\sigma(2.0)=0.8808$ — (1) $i_t=\sigma(0.0)=0.5000$ — (1) $o_t=\sigma(1.0)=0.7311$ — (1) $\tilde c_t=\tanh(-1.0)=-0.7616$ — (1)

(b) $c_t = f_t c_{t-1}+i_t\tilde c_t = 0.8808(0.8)+0.5(-0.7616)=0.7046-0.3808=0.3238$ — (2) $h_t = o_t\tanh(c_t)=0.7311\cdot\tanh(0.3238)=0.7311\cdot0.3129=0.2288$ — (2)

(c) With $f_t\approx1,\ i_t\approx0$ : $c_t\approx c_{t-1}$ , so the cell state is copied nearly unchanged forward (a "constant error carousel"). — (2) The gradient of $c_t$ w.r.t. $c_{t-1}$ is $\approx f_t\approx 1$ , so backpropagated gradient through the cell path is multiplied by ~1 each step instead of by a factor $<1$ (as in the $\tanh$ Jacobian of vanilla RNNs). — (2) Hence gradients do not shrink exponentially over long time lags, letting the LSTM learn long-range dependencies. — (2)

Question 3 (12 marks)

(a) $\text{score}_1 = [1,1]\cdot[1,0]=1$ ; $\text{score}_2=[1,1]\cdot[0,1]=1$ ; $\text{score}_3=[1,1]\cdot[1,1]=2$ . (3)

(b) $e^1=2.71828,\ e^1=2.71828,\ e^2=7.38906$ ; sum $=12.82562$ . (2) $\alpha_1=\alpha_2=2.71828/12.82562=0.2119$ ; $\alpha_3=7.38906/12.82562=0.5761$ . (3) (check: $0.2119+0.2119+0.5761=0.9999\approx1$ )

(c) $c=\alpha_1[1,0]+\alpha_2[0,1]+\alpha_3[1,1]$ $c_x=0.2119+0.5761=0.7881$ ; $c_y=0.2119+0.5761=0.7881$ . $c=[0.7881,\ 0.7881]$ . (4)

Question 4 (10 marks)

(a) Top-2 of step 1: "A" ( $-0.5$ ), "B" ( $-1.0$ ). "" $(-3.0)$ dropped. (3)

(b) Expand (cumulative = prefix + step2):

A→A: $-0.5-1.2=-1.7$
A→B: $-0.5-0.7=-1.2$
A→: $-0.5-2.0=-2.5$
B→A: $-1.0-0.4=-1.4$
B→B: $-1.0-1.5=-2.5$
B→: $-1.0-1.0=-2.0$ (4)

(c) Rank all: A→B $(-1.2)$ , B→A $(-1.4)$ , A→A $(-1.7)$ , B→ $(-2.0)$ , ... Top-2 surviving: A→B ( $-1.2$ ) and B→A ( $-1.4$ ). Top sequence: A→B. (3)

Question 5 (10 marks)

(a) Teacher forcing always feeds the ground-truth previous token during training, but at inference the model feeds its own (possibly wrong) predictions. The model never learned to recover from its own errors ⇒ errors compound (exposure bias). Mitigation: scheduled sampling (gradually replace ground-truth with model predictions during training), or beam search / minimum-risk / sequence-level training. (4) (2 for problem, 2 for mitigation)

(b) Pad all sequences in a batch to the max length (or bucket by similar length to reduce padding), use a mask marking real vs pad tokens; the loss is computed only over non-pad positions (mask-weighted) so padding does not contribute spurious gradients/loss. (3)

(c) Encoder sees the whole input at once, so a bidirectional pass lets each encoded state incorporate both past and future context ⇒ richer representations. The decoder generates tokens left-to-right and cannot see future outputs at inference (they don't exist yet), so a backward pass is impossible/causally invalid. (3)

[
  {"claim":"Q1a h1 and h2 values", "code":"import sympy as sp; h1=sp.tanh(1.0); h2=sp.tanh(0.5*h1-0.5); result = (round(float(h1),4)==0.7616 and round(float(h2),4)==-0.1186)"},
  {"claim":"Q1b dL/dWh = -0.0891", "code":"import sympy as sp; h1=sp.tanh(1.0); h2=sp.tanh(0.5*h1-0.5); g=h2*(1-h2**2)*h1; result = round(float(g),4)==-0.0891"},
  {"claim":"Q2 LSTM c_t and h_t", "code":"import sympy as sp; f=sp.sigmoid(2.0); i=sp.sigmoid(0.0); o=sp.sigmoid(1.0); ct=f*0.8+i*sp.tanh(-1.0); ht=o*sp.tanh(ct); result = (round(float(ct),4)==0.3238 and round(float(ht),4)==0.2288)"},
  {"claim":"Q3 attention weights and context", "code":"import sympy as sp; import math; s=[math.exp(1),math.exp(1),math.exp(2)]; Z=sum(s); a=[x/Z for x in s]; cx=a[0]*1+a[1]*0+a[2]*1; cy=a[0]*0+a[1]*1+a[2]*1; result = (round(a[2],4)==0.5761 and round(a[0],4)==0.2119 and round(cx,4)==0.7881 and round(cy,4)==0.7881)"},
  {"claim":"Q4 beam survivors A->B and B->A top scores", "code":"cands={'AA':-1.7,'AB':-1.2,'Ae':-2.5,'BA':-1.4,'BB':-2.5,'Be':-2.0}; top2=sorted(cands.items(), key=lambda kv:-kv[1])[:2]; result = ([k for k,v in top2]==['AB','BA'])"}
]