Question bank — Sequence-to-sequence models

False — the encoder/decoder split makes $T_x$ and $T_y$ independent; the decoder keeps emitting tokens until it produces <EOS>, regardless of input length.

True — vanilla seq2seq discards $h_1^{(enc)},\dots ,h_{T_x-1}^{(enc)}$ and keeps only $\mathbf{c}=h_{T_x}^{(enc)}$; reusing the discarded states is exactly what attention adds.

False — teacher forcing feeds the ground-truth previous token, which only exists during training; at inference the decoder must feed its own prediction back in (see figure s02).

False — a larger $k$ explores more paths so it usually finds an equal-or-better scoring sequence, but there is no guarantee of the global optimum; only exhaustive search would, and even that maximizes model probability, not translation quality.

True — the decoder predicts target-language tokens, so $W_o{h}_t^{(dec)}+b_o$ has $|V_{\text{target}}|$ outputs and softmax normalizes across those.

False — in the basic model $\mathbf{c}$ is computed once and held fixed for the whole decode; only attention recomputes a fresh context vector per decoding step.

False — $\mathbf{c}$ seeds the decoder's memory as $h_0^{(dec)}=\mathbf{c}$ (figure s01), and in many variants is also concatenated to every decoder input, so its influence persists across the whole output.

True — the sequence probability is a product ${\prod }_{t}P(y_t\mid \dots )$, and taking $-\log$ turns a product into a sum, giving $\mathcal{L}=-{\sum }_t\log P(y_t^{\ast }\mid y_{<t}^{\ast },\mathbf{c})$; the derivation of this equivalence lives in cross-entropy loss.

True — with a single beam, beam search keeps only the top-1 partial sequence at every step, which is exactly the argmax-at-each-step rule of greedy decoding.

False — the decoder generates left-to-right and cannot see future output tokens (they don't exist yet), so it stays unidirectional; only the encoder, which sees the whole input at once, can be bidirectional.

False — padding is masked out: the loss ignores padded positions and masking stops padded steps from polluting $\mathbf{c}$, otherwise every batch would learn from meaningless filler.

Wrong — decoding length is controlled by the decoder emitting <EOS> (or hitting a max-length cap), not by input length; the input was already fully consumed during encoding.

Wrong reason — softmax is used because we must pick one token from $|V|$ classes and therefore need a valid probability distribution; speed is not the motivation.

Wrong — under teacher forcing the ground-truth token is fed at $t+1$ regardless of the prediction (figure s02); feeding the model's own token is inference behaviour, and that gap causes exposure bias.

Backwards — each token contributes a log-probability, and since a probability is $\le 1$ its log is $\le 0$; a longer sequence therefore adds more negative terms, giving a smaller (more negative) raw sum. So raw sums penalize length, which is why length normalization $\frac{1}{t}{\sum }_i\log P(y_i\mid \dots )$ divides by the length $t$ to level the field.

Wrong — $\mathbf{c}\in {\mathbb{R}}^{d_h}$ has the encoder's hidden dimension $d_h$, unrelated to $|V|$; the vocabulary size only appears at the final softmax layer's output width via $W_o$.

Wrong — gradients flow from the loss back through the decoder, into $\mathbf{c}$, and then through the entire encoder; BPTT spans both RNNs, as covered in backpropagation through time.

Wrong — they are two distinct vocabulary entries: <SOS> is the decoder's initial input to kick off generation, <EOS> is a predicted output that signals stopping.

Because input length $T_x$ varies per example, and only a recurrent structure (3.5.01-Recurrent-neural-networks, 3.5.05-LSTM-networks, 3.5.06-GRU-networks) can consume an arbitrary-length sequence with one fixed set of weights.

Because all information is squeezed into one fixed-size $\mathbf{c}$; for long inputs this is an information bottleneck and early tokens get forgotten by the time the encoder reaches $h_{T_x}^{(enc)}$.

Because $\log$ is monotonically increasing, so maximizing $\log P$ maximizes $P$; flipping the sign turns "maximize" into "minimize", giving a loss the optimizer can descend.

Because at training the decoder always sees correct previous tokens (teacher forcing, figure s02), so it never learns to recover from its own mistakes, yet at inference it must feed its own possibly-wrong tokens — the two regimes present different input distributions.

Because greedy commits to the single best token each step and can walk into a dead-end, while beam search keeps $k$ hypotheses alive so a locally-suboptimal early token can still lead to a globally better sequence (details in 4.2.01-Beam-search).

Because padding just fills short sequences up to the batch's max length; without a mask the encoder would treat filler as real tokens and the loss would be computed on meaningless positions, corrupting both $\mathbf{c}$ and the gradients.

Because attention lets the decoder build a fresh context each step from all encoder states, so no single vector must memorize the whole input — the bottleneck disappears (3.5.08-Attention-mechanism, taken further in 3.5.09-Transformers).

Because a poorly-trained decoder might never emit <EOS> and loop forever; the length cap is a safety net guaranteeing termination.

It works fine: the encoder runs one step to produce $\mathbf{c}=h_1^{(enc)}$, and the decoder generates tokens until <EOS>; there is no requirement that $T_x=T_y$.

The output sequence is empty (length zero) — a valid, if usually undesirable, outcome; the model has decided the correct translation is "nothing".

That completed hypothesis is frozen with its length-normalized score (figure s03); the remaining beams keep expanding, and at the very end the best frozen hypothesis across all finish-times is chosen.

The longer one wins, because dividing the same total by a larger length $t$ gives a higher (less negative) per-token average — the length-normalization effect.

At least <SOS>, <EOS>, and one real token; without <EOS> it could never signal termination and without <SOS> the decoder would have no initial input.

They can vanish or explode across many encoder time steps (the classic BPTT problem, 3.4.03-Backpropagation-through-time); LSTM/GRU gating mitigates but does not fully cure this.

The 9 padded steps are masked so they neither update $\mathbf{c}$ nor add loss; the encoder's effective summary for that sentence is taken at its true last step (position 3), not at the padded position 12.

Early in training predictions are near-random, so errors compound step-by-step and the decoder rarely sees correct context, making learning slow and unstable — teacher forcing exists to avoid exactly this.