Handling variable-length sequences: Question bank

This is a stress-test for your understanding of handling variable-length sequences. Each line below is a Question ::: Answer reveal. Cover the answer, commit to a response and a reason, then check. If your reason is wrong even when your yes/no is right, you found a real gap.

Before we start, one shared vocabulary reminder so nothing here is a mystery.

Definition Every symbol used on this page, defined once

Padding = adding filler tokens (usually the value $0$ or a <PAD> symbol) to the end of a short sequence so every sequence in a group has the same length.
Mask = a companion array of $1$ s and $0$ s: $1$ marks a real token, $0$ marks a pad token. It is the model's way of remembering which positions are fake.
Batch = a stack of sequences processed together as one rectangular block for speed. See Batch Processing.
$B$ = number of sequences in the batch. We index a sequence by $i \in {1, \dots, B}$ .
$t$ = the time step (position) inside a sequence, $t \in {1, \dots, L_{ma x}}$ .
$L_{i}$ = the true length of sequence $i$ (real tokens only, no pads).
$L_{ma x} = max_{i} L_{i}$ = the length we pad every sequence up to, so the batch is rectangular.
$m_{i} (t)$ = the mask entry for sequence $i$ at position $t$ : it is $1$ when $t \leq L_{i}$ (real token) and $0$ when $t > L_{i}$ (pad token).
$ℓ_{i} (t)$ = the per-position loss (e.g. cross-entropy) for sequence $i$ at position $t$ .
$L$ = whenever we discuss a single sequence in isolation (e.g. attention), $L$ is that one sequence's true length; in a batch it equals $L_{i}$ for that row.

Definition Attention vocabulary used below, in plain words

Attention answers "for a position that is asking a question, how much should it listen to each other position?" It uses three named vectors:

query $q$ = the vector for the position doing the asking — "what am I looking for?".
key $k_{j}$ = the vector for position $j$ advertising what it offers — "here is what I am about". The score $q^{⊤} k_{j}$ (a dot product) measures how well the query matches key $j$ .
value $v_{j}$ = the vector position $j$ actually hands back if it gets attended to — "here is the content I contribute". The attention output is the weighted sum $\sum_{j} α_{j} v_{j}$ , where $α_{j}$ is the softmax weight for position $j$ .
$d_{k}$ = the length (dimension) of the query and key vectors — how many numbers each one has. It shows up when we scale scores by $\frac{1}{d _{k}}$ .

The recurring trap of this whole topic is this: padding is a storage convenience, but the maths must behave as if the padding were never there. Almost every item below is a variation on that one idea.

The two figures below carry the visual intuition that the reveals refer back to — look at them first.

Figure 1 — what padding and masking actually look like on a batch.

Figure — Handling variable-length sequences

Each row is one sequence, padded up to $L_{ma x} = 5$ . Green cells are real tokens ( $m_{i} (t) = 1$ ), gray cells are pads ( $m_{i} (t) = 0$ ). The mask on the right is the exact same grid written as $1$ s and $0$ s — that is all a mask is.

Figure 2 — why $- \infty$ , not $0$ , is the correct mask for softmax attention.

The left bars are raw attention scores over three keys where the third is a pad. The middle panel shows the wrong fix (multiply the score by $0$ ): after softmax the pad still steals weight. The right panel shows the correct fix (add $- \infty$ before softmax): the pad's weight collapses to zero and the two real weights renormalise to sum to $1$ . Keep this picture in mind for SE3, SE4 and EC6.

True or false — justify

TF1. "Padding changes what the model outputs at the real (non-pad) positions."

False if masking is done correctly. Pads exist only to make the tensor rectangular; a correct mask (Figure 1) zeroes their contribution to loss and attention, so real positions get identical results with or without them.

TF2. "A longer sequence should always have a larger loss than a shorter one."

False. Loss quality is per-token. We divide by the mask sum

\sum_{t} m_{i} (t) = L_{i}

(the true length) so loss is a per-token average — a 3-word sentence and a 40-word one are compared fairly, not penalised by length.

TF3. "Attention needs padding baked into its formula to work on variable lengths."

False. For a single sequence of true length

L

, the sum

\sum_{j = 1}^{L} α_{j} v_{j}

— where

v_{j}

is the value vector at position

j

and

α_{j}

its softmax weight — already runs over exactly its real tokens. Padding only enters when we batch (raising the row length to

L_{ma x}

), and we then neutralise it with a

- \infty

mask (Figure 2), never inside the core formula. See Attention Mechanism.

TF4. "Packing a sequence with pack_padded_sequence gives a different final hidden state than running the padded sequence through the RNN."

False (with correct masking). Packing is a speed optimisation that skips computing over pads; the RNN visits exactly the same real tokens in the same order, so the hidden states at real positions match. See Recurrent Neural Networks.

TF5. "Using

0

as the pad token is safe because zeros don't affect a weighted sum."

False. A zero input still produces a non-zero hidden state through the bias and

tanh

, and a zero embedding can still receive attention weight. Zeros are harmless only when a mask explicitly removes them — the value

0

alone is not a shield.

TF6. "In self-attention, masking only matters for the keys, not the queries."

Half-true, context-dependent. For ignoring pad positions being attended to (the keys — the positions being looked at), we mask keys. But we usually also skip loss on pad query positions (the positions doing the asking). Both matter; which one you mask depends on whether the pad is what you look at or what you predict for.

TF7. "Dynamic unrolling and dynamic control flow mean the model has different weights for different lengths."

False. The whole point is parameter sharing: the same

W_{hh}, W_{x h}, b_{h}

are reused at every time step and for every length. "Dynamic" refers to how many steps run, not which weights.

TF8. "Transformers, unlike RNNs, don't need any masking for padding."

False. Transformers still pad to a common length

L_{ma x}

for batching, and still need a padding mask to stop attention from spending weight on pad tokens. They additionally use a causal mask in decoders — masking is arguably more central to them, not less. See Masking in Deep Learning.

TF9. "In multi-head attention the padding mask must be recomputed separately for each head."

False. The padding mask depends only on which positions are pads, not on the head. One mask of shape (batch, key-positions) is broadcast across all heads and all query positions. The Q/K/V (query/key/value) projections differ per head, but they never move a pad token to a non-pad slot, so the same mask is valid everywhere.

Spot the error

SE1. "To pad a batch, I extend every sequence to length 100 (a fixed constant I chose in advance)."

Error: pad to the batch's longest sequence $L_{ma x}$ , not a global constant. A fixed 100 wastes memory when the batch's max is 12, and breaks when a sequence exceeds 100. Pad length should be dynamic per batch.

SE2. "My masked loss is

L_{i} = \frac{1}{L _{ma x}} \sum_{t = 1}^{L_{ma x}} m_{i} (t) ℓ_{i} (t)

Error: dividing by $L_{ma x}$ instead of $\sum_{t} m_{i} (t) = L_{i}$ . The correct form is

L_{i} = \frac{1}{\sum _{t} m _{i} ( t )} \sum_{t} m_{i} (t) ℓ_{i} (t)

; dividing by the padded length

L_{ma x}

shrinks short-sequence losses artificially, so the model under-learns from short examples.

SE3. "Before softmax I multiply the pad scores by

0

to remove them."

Error: you must add $- \infty$ , not multiply by $0$ (Figure 2, middle vs right). A score of

0

still gives

e^{0} = 1

, a real chunk of attention; only

e^{- \infty} = 0

truly removes it while letting the denominator renormalise.

SE4. "I applied softmax first, then zeroed the pad weights afterward."

Error: order is wrong and the result no longer sums to 1. Zeroing after softmax leaves the surviving weights summing to less than 1, silently scaling down the output. Mask the scores before softmax so normalisation still divides by the correct total (Figure 2, right).

SE5. "I packed my batch but forgot to sort sequences by length — PyTorch's older API happily accepted it."

Error: classic pack_padded_sequence (without enforce_sorted=False) assumes descending length order. Unsorted input scrambles batch_sizes, so the RNN pairs the wrong tokens with the wrong hidden states.

SE6. "I masked the loss but let padding flow through the RNN and used the last time step's hidden state as my sentence summary."

Error: the "last" step for a padded sequence is a pad step. You must gather the hidden state at index

L_{i} - 1

(the last real token), or the summary is contaminated by pads.

SE7. "Bidirectional RNN over a padded batch — I just read left-to-right and right-to-left as normal."

Error: the backward pass starts at the padding. Without masking/packing, the reverse direction begins by encoding pad tokens into the real ones. Bi-RNNs need packing or careful masking on both directions. See LSTM and GRU.

SE8. "To be safe I used the literal float('-inf') for my mask value in half-precision (fp16)."

Error: on GPUs/TPUs a literal $- \infty$ produces NaN the moment a row is fully masked (softmax of all $- \infty$ is $\frac{0}{0}$ ), and $- \infty$ also breaks fp16 arithmetic elsewhere. Practitioners use a large finite negative number instead — e.g. add roughly

- 1 0^{9}

in fp32 or the smallest finite fp16 value — so

e^{score}

underflows to

0

without ever forming a true

\infty

Why questions

WHY1. Why do we bother padding at all instead of feeding each sequence one at a time?

Because hardware (GPUs) is fastest on rectangular blocks processed in parallel. One-at-a-time is correct but slow; padding lets a whole batch flow through together.

WHY2. Why divide the masked loss by the mask sum rather than a constant?

So the loss is a genuine per-real-token average. The mask sum

\sum_{t} m_{i} (t)

equals the true length

L_{i}

, making every sequence contribute on equal footing regardless of how much padding it carries.

WHY3. Why does attention scale scores by

\frac{1}{d _{k}}

, and how does this relate to variable lengths?

Here

d_{k}

is the query/key vector dimension (defined above). Scaling keeps the score variance near

1

so softmax doesn't saturate — it depends on the feature dimension

d_{k}

, not on the length. And that is exactly the point for variable lengths: because the stabiliser is length-free, the same attention block behaves identically whether

L = 5

L = 500

, so no per-length retuning is needed. (Length only ever enters through masking, never through the scaling.)

WHY4. Why is packing a "memory and compute" win, while dynamic unrolling is only a compute win?

Packing physically stores no pad values (less memory) and skips computing over them (less compute). Dynamic unrolling still holds the padded tensor in memory; it merely avoids stepping past each sequence's true end.

WHY5. Why can <PAD> sometimes get a learned embedding, yet we still must mask it?

A learned pad embedding is convenient for a uniform lookup table, but it still carries a real vector that attention/loss would otherwise use. The embedding's existence doesn't excuse it from being masked out of the maths.

WHY6. Why does attention have quadratic complexity in length while an RNN is linear?

Attention compares every position with every other (

L \times L

scores), giving

L^{2}

. An RNN walks the sequence once, one step per token, giving

L

. This trade-off is central when choosing between seq2seq RNNs and transformer stacks.

WHY7. Why do we prefer masking over simply removing pad rows from the tensor mid-computation?

Removing rows makes the tensor ragged again, defeating the rectangular-batch speed win. Masking keeps the shape fixed while neutralising the pads' effect — best of both worlds.

Edge cases

EC1. What is the mask, loss, and RNN behaviour for a sequence of length

1

in a batch padded to

L_{ma x} = 50

Mask

m (t)

[1, 0, 0, \dots, 0]

; loss divides by

\sum_{t} m (t) = 1

(only the real token counts); the RNN produces one meaningful hidden state and 49 pad steps that must be masked/packed away.

EC2. What happens if an entire sequence is empty (

L_{i} = 0

The mask sum is

0

, so the per-sequence loss

\frac{1}{0} \sum \dots

divides by zero. Handle it explicitly — drop empty sequences before batching, or clamp the denominator to

max (1, \sum_{t} m_{i} (t))

— never let a zero-length row reach the loss.

EC3. If every sequence in a batch has the same length (

L_{i} = L_{ma x}

for all

i

), do we still need masks?

No padding means no pad positions, so the mask is all

1

s and does nothing. Masking is harmless but redundant here — the machinery only earns its keep when lengths differ.

EC4. A batch has one length-500 sequence among many length-5s. What does padding cost, and how does bucketing help — with numbers?

Padding forces everyone to

L_{ma x} = 500

: a batch of 32 stores

32 \times 500 = 16000

token-slots for only

\approx 500 + 31 \times 5 = 655

real ones — about 96% wasted. Bucketing groups similar lengths so the length-5s pad only to

\sim 5

(

32 \times 5 = 160

slots, near-zero waste) and the length-500 sits in its own bucket. Trade-off: buckets that are too fine hurt batch parallelism and shuffling randomness, so you tune bucket width to balance pad-waste against throughput. See Batch Processing.

EC5. In a causal (autoregressive) decoder, a position is masked from the future and from padding. What does the combined mask allow at the last real token?

It permits attention to all earlier real tokens, forbids all future positions, and forbids all pad positions. Concretely, the combined mask is the intersection of two constraints — the lower-triangular causal mask (no looking ahead) and the padding mask (no looking at pads) — so a position may attend only where both say "allowed". See Masking in Deep Learning.

EC6. If a query position is itself a pad token, does masking its keys make its output valid?

No — its output is meaningless regardless, because the query vector is a pad. We simply don't use the outputs at pad query positions (loss is masked there), so their garbage never propagates.

EC7. Attention over a sequence where all keys are masked (a fully-pad row) — what does softmax return, and how do frameworks survive it?

Softmax of all

- \infty

\frac{0}{0}

— NaN (Figure 2 shows why the denominator must stay positive). Safeguards: keep at least one unmasked position, add a finite large-negative instead of true

- \infty

, or special-case fully-pad rows to output zeros. This is the same NaN trap as SE8.

Recall One-line summary to lock in

Padding is a shape trick; masking is the promise that the shape trick never changes the answer. Answer ::: Every trap on this page is a case where padding was allowed to leak into the maths — the fix is always to mask (add $- \infty$ / weight by the mask / divide by the true length $L_{i}$ ), never to trust that $0$ -valued pads are automatically harmless.