Sequence Models

Q1. (4 marks) $h_t = \tanh(W_{hh} h_{t-1} + W_{xh} x_t + b_h)$ $y_t = W_{hy} h_t + b_y$

• $W_{xh}$: input-to-hidden weights; $W_{hh}$: hidden-to-hidden (recurrent) weights; $W_{hy}$: hidden-to-output weights; $b_h, b_y$: biases. (1 for each correct equation = 2; 2 for correct definitions of matrices/bias) Why: the recurrent term $W_{hh}h_{t-1}$ carries memory of past inputs.

Q2. (4 marks)

• Statement: during BPTT, gradients propagated back over many time steps become exponentially small, so early time steps receive negligible weight updates → the network fails to learn long-range dependencies. (2)
• Reason: the gradient contains a product of Jacobians $\prod_k \frac{\partial h_k}{\partial h_{k-1}}$; each factor $\approx W_{hh}^\top \text{diag}(\tanh')$. Since $|\tanh'|\le 1$ and repeated multiplication of terms $<1$ decays geometrically, the product $\to 0$. (2)

Q3. (4 marks) — 1 mark each gate + 1 for completeness/correctness:

• Forget gate $f_t$: decides what fraction of the old cell state to keep/discard.
• Input gate $i_t$: decides what new candidate information to add to the cell state.
• Output gate $o_t$: decides what part of the cell state is exposed as the hidden state $h_t$.

Q4. (4 marks)

• Gates: update gate $z_t$ and reset gate $r_t$. (2) $h_t = (1 - z_t)\odot h_{t-1} + z_t \odot \tilde{h}_t$ (2) Why: $z_t$ interpolates between keeping the old state and taking the new candidate — an adaptive memory mixing. (Accept the equivalent convention $h_t = z_t h_{t-1} + (1-z_t)\tilde h_t$ if stated consistently.)

Q5. (4 marks)

• Unidirectional RNN processes the sequence in one direction (past→future), so $h_t$ depends only on inputs up to $t$. Bidirectional RNN runs two RNNs (forward + backward) and concatenates their hidden states, so each output depends on both past and future context. (2)
• Cannot be used for real-time / online / streaming generation (e.g., live speech-to-text output, autoregressive text generation) because the future part of the sequence is not yet available. (2)

Q6. (4 marks)

• Encoder RNN reads the input sequence and compresses it into a fixed-length context vector (its final hidden state). The decoder RNN is initialized with this context and generates the output sequence step by step. (2)
• Context vector = the summarized representation of the entire input. (1)
• Limitation: a single fixed-length vector becomes an information bottleneck for long inputs — performance degrades as input length grows (motivating attention). (1)

Q7. (4 marks)

• Teacher forcing: during training, the decoder receives the ground-truth previous token as input (instead of its own predicted token) when predicting the next token. (2)
• Advantage: faster, more stable convergence. (1)
• Disadvantage: exposure bias — mismatch between training (true tokens) and inference (predicted tokens) can hurt test-time performance. (1)

Q8. (4 marks) Softmax of $e=[2,1,0]$: $\alpha_i = \frac{e^{e_i}}{\sum_j e^{e_j}}, \quad \sum = e^2+e^1+e^0 = 7.389+2.718+1 = 11.107$

• $\alpha_1 = 7.389/11.107 = 0.665$
• $\alpha_2 = 2.718/11.107 = 0.245$
• $\alpha_3 = 1/11.107 = 0.090$ (2 for method, 2 for correct values ≈ [0.665, 0.245, 0.090])

Q9. (4 marks)

• Beam search keeps the top-$k$ highest-probability partial sequences ("beams") at each decoding step, expanding all of them and retaining the best $k$ overall. (2)
• Difference: greedy decoding ($k=1$) picks only the single most probable token each step and cannot recover from an early mistake; beam search explores multiple hypotheses. (1)
• Candidates scored per step: $k \times V$. (1)

Q10. (4 marks)

• A word embedding is a dense, low-dimensional real-valued vector representing a word such that semantically similar words are close in vector space. (2)
• Word2Vec variants: CBOW (predicts the target/center word from surrounding context words) and Skip-gram (predicts surrounding context words from the target word). (2)

[
  {"claim":"Softmax weights of [2,1,0] round to [0.665,0.245,0.090] and sum to 1",
   "code":"import math\ne=[2,1,0]\nd=sum(math.exp(x) for x in e)\na=[round(math.exp(x)/d,3) for x in e]\nresult = (a==[0.665,0.245,0.090]) and (round(sum(math.exp(x)/d for x in e),6)==1.0)"},
  {"claim":"Denominator e^2+e^1+e^0 approx 11.107",
   "code":"import math\nresult = round(math.exp(2)+math.exp(1)+math.exp(0),3)==11.107"},
  {"claim":"Beam search scores k*V candidates per step for k=3,V=1000 -> 3000",
   "code":"k=3; V=1000\nresult = k*V==3000"},
  {"claim":"GRU convex combination: for z=0.7, h=(1-z)*h_prev+z*hcand with h_prev=2,hcand=5 equals 4.1",
   "code":"z=0.7; hp=2; hc=5\nresult = round((1-z)*hp+z*hc,6)==4.1"}
]