Handling variable-length sequences: Worked examples

This page is the worked-example gauntlet for Handling variable-length sequences. The parent note gave you the machinery: masking, packing, dynamic unrolling, and attention. Here we drive that machinery through every case it can hit — every length pattern, every degenerate input, the limiting cases, a real word problem, and an exam twist.

Before we compute anything, one promise: every symbol is earned before it is used. If a term shows up, it was defined in the parent note or is defined here in plain words first.

The scenario matrix

Think of "variable-length handling" as a machine with dials. Each cell below is a distinct way the world sets those dials. If we work one example per cell, the reader never meets an unshown situation.

#	Case class	What is unusual about it	Covered by
C1	Ordinary batch — several sequences, all different lengths	The baseline: pad → mask → normalise	Example 1
C2	Masked loss normalisation	Prove shorter ≠ artificially cheaper loss	Example 2
C3	Degenerate: length-1 sequence	Almost all of the row is padding	Example 3
C4	Degenerate: all sequences same length	Padding does nothing — must still be correct	Example 4
C5	Packing bookkeeping	Reconstruct `batch_sizes` from lengths and back	Example 5
C6	Attention + `-∞` mask	Softmax must send padding weight to exactly 0	Example 6
C7	Limiting case: score scale $d_{k}$	What happens as dimension $d_{k}$ grows	Example 7
C8	Real-world word problem — chatbot batch cost	Turn padding waste into a number	Example 8
C9	Exam twist — unsorted lengths break packing	Spot the bug, sort, recover	Example 9

Notation we will reuse (all defined here)

Definition The seven quantities

$B$ = number of sequences in a batch (the batch size).
$L_{i}$ = the true length of sequence $i$ (real tokens, no padding).
$L_{m a x} = max_{i} L_{i}$ = the length we pad every row up to.
$L_{avg} = \frac{1}{B} \sum_{i} L_{i}$ = the average true length across the batch (used only in the cost word problem).
$m_{i} (t)$ = the mask: $1$ if position $t$ is a real token, $0$ if it is padding.
$y_{i} (t)$ = the true target token at position $t$ of sequence $i$ (the correct answer the model should produce).
$\overset{y}{^}_{i} (t)$ = the model's prediction at position $t$ of sequence $i$ (what it actually produced — a probability vector over the vocabulary).
$ℓ (y_{i} (t), \overset{y}{^}_{i} (t))$ = the per-token loss: a single number measuring how far the prediction $\overset{y}{^}_{i} (t)$ is from the truth $y_{i} (t)$ at one position (e.g. cross-entropy). Small when the model is right, large when wrong.

Everything below is arithmetic on these seven.

Example 1 — C1: the ordinary batch

Forecast: guess the total number of cells and the fraction that is padding before reading on.

Find $L_{m a x}$ . $L_{m a x} = max (3, 4, 1) = 4$ . Why this step? The batch tensor must be rectangular, so every row is stretched to the longest row.
Pad each row to width 4. $[1, 5, 7] [2, 8, 3, 9] [4] \to [1, 5, 7, 0] \to [2, 8, 3, 9] \to [4, 0, 0, 0]$ Why this step? Appending $0$ (the <PAD> id) fills the slots without inventing meaning — as long as we later mask them.
Build the mask $m_{i} (t)$ (1 = real, 0 = pad), positions $t = 1, \dots, 4$ : $M = 111110110010$ Why this step? The mask is the ground truth of which cells count; see the picture below.
Count. Total cells $= B \cdot L_{m a x} = 3 \cdot 4 = 12$ . Real cells $= \sum_{i} L_{i} = 3 + 4 + 1 = 8$ . Waste $= 12 - 8 = 4$ .

Figure below (s01): the $3 \times 4$ padded grid drawn in blueprint style — real token cells are cyan (labelled with their token id), padding cells are outlined in amber and labelled "PAD". An amber arrow points to the padding block (top-right), a cyan arrow to the real tokens, and the true length $L_{i}$ is annotated at the end of each row. It is the picture of "8 real, 4 wasted".

Figure — Handling variable-length sequences

Verify: waste fraction $= 4/12 = 1/3 \approx 33.3%$ . Sanity: only row 3 (the short "AI") should waste much — it contributes $4 - 1 = 3$ of the $4$ wasted cells, and row 1 contributes $1$ . $3 + 1 = 4$ . ✓

Example 2 — C2: why we divide by the mask sum

Forecast: which version makes the short sentence look "better" than it deserves?

Normalised (correct) loss. $L_{1} = \frac{1}{\sum _{t} m _{1} ( t )} \sum_{t} m_{1} (t) ℓ = \frac{1}{3} (2 + 2 + 2) = 2.0$ Why this step? Dividing by $\sum_{t} m_{1} (t) = L_{1} = 3$ averages over real tokens only.
Wrong version — divide by $L_{m a x} = 4$ . $L_{1}^{wrong} = \frac{1}{4} (2 + 2 + 2 + 0) = \frac{6}{4} = 1.5$ Why this step? The padding term $0 \cdot ℓ = 0$ still occupies a slot in the denominator, diluting the average.
Interpret. The wrong number $1.5 < 2.0$ pretends the model did better on the short sentence purely because it is short. Why this step? Optimisers chase lower loss — a length-dependent bias would push the model to favour short sequences.

Verify: with true per-token loss exactly $2.0$ , the correct average must equal $2.0$ regardless of length, while the un-normalised form gives $1.5$ . Since $1.5 \neq = 2.0$ , the un-normalised form is provably biased. ✓ (This is the algebra behind the parent's "shorter sequences would have artificially lower losses".)

Example 3 — C3: the length-1 degenerate row

Forecast: the short row is almost entirely padding — what does that do to its per-token weight?

Mask row for sequence 2, padded to $L_{m a x} = 6$ , positions $t = 1, \dots, 6$ : $m_{2} = [1, 0, 0, 0, 0, 0]$ Why this step? Only position $t = 1$ is real; positions $t = 2, \dots, 6$ are pad.
Per-token weight. Its loss is $L_{2} = \frac{1}{\sum _{t} m _{2} ( t )} \sum_{t} m_{2} (t) ℓ = \frac{1}{1} ℓ (y_{2} (1), \overset{y}{^}_{2} (1))$ , where $y_{2} (1)$ is the true token and $\overset{y}{^}_{2} (1)$ the model's prediction at the single real position (both defined in the notation box). Why this step? The single real token carries all the weight $= 1$ ; the normaliser is $\sum_{t} m_{2} = 1$ , so no dilution.
Waste fraction of that row. Real $= 1$ , padded width $= 6$ , waste $= 5/6 \approx 83.3%$ . Why this step? This is exactly why packing exists — dynamic unrolling / packing would skip those five dead cells entirely.

Verify: the normaliser can never be $0$ here because $L_{i} \geq 1$ for any real sequence, so division is always safe. Row waste $5/6 = 0.8333 \dots$ ✓

Common mistake The empty sequence — how to guard it

If $L_{i} = 0$ ever occurs (a truly empty row), $\sum_{t} m_{i} (t) = 0$ and $L_{i} = \frac{0}{0}$ — undefined. Concretely, frameworks filter empty rows out of the batch before padding:

# keep only non-empty sequences, remember their original positions
keep_idx  = [i for i, L in enumerate(lengths) if L > 0]
sequences = [sequences[i] for i in keep_idx]
lengths   = [lengths[i]   for i in keep_idx]
# ... pad / pack / run the model on the filtered batch ...
# scatter outputs back into a full-size tensor using keep_idx,
# writing a neutral value (e.g. zeros) for the dropped empty rows

A defensive alternative is to clamp the denominator: replace $\sum_{t} m_{i} (t)$ with $max (\sum_{t} m_{i} (t), 1)$ so the division never blows up, then multiply that row's contribution by $0$ . Either way, an empty row must contribute zero loss, never $\frac{0}{0}$ .

Example 4 — C4: all lengths equal (padding does nothing)

Forecast: if nothing needs padding, does the masked-loss formula still give the plain average?

$L_{m a x} = max (5, 5, 5) = 5$ ; no row is stretched. Why this step? Padding is defined relative to the longest row — here every row is the longest.
Mask is all ones (positions $t = 1, \dots, 5$ ): $M = 111111111111111$ Why this step? Every position is real, so no zeros.
Masked loss collapses to the plain mean. For each row $L_{i} = \frac{1}{5} \sum_{t = 1}^{5} ℓ$ . Why this step? With $m_{i} (t) \equiv 1$ the mask factors out — proving our formula degrades gracefully to the ordinary average.

Verify: waste $= B L_{m a x} - \sum_{i} L_{i} = 15 - 15 = 0$ . ✓ The general formula must reduce to the un-masked one in this limit, and it does.

Example 5 — C5: packing bookkeeping both directions

Forecast: batch_sizes counts sequences still "alive" at each step — sketch that count in your head.

Recall our fixed 1-based convention: columns are $t = 1, \dots, L_{m a x}$ , and a row of true length $L_{i}$ is alive exactly at $t = 1, \dots, L_{i}$ .

Draw the padded grid (rows = sequences a,b,c; columns = time $t = 1 \dots 4$ ):

Figure below (s02): the same blueprint grid, but now each real cell is labelled with its token name and 1-based time index ( $a 1, a 2, a 3, a 4$ for the length-4 row; $b 1, b 2, b 3$ ; just $c 1$ ). Dead cells are amber dashes. Along the bottom, amber numbers give the alive count for each column — that row of numbers is the batch_sizes array $[3, 2, 2, 1]$ .

Count living sequences per column. At each time $t$ , how many rows still have a real token (i.e. $L_{i} \geq t$ )? $t = 1 : 3, t = 2 : 2, t = 3 : 2, t = 4 : 1$ Why this step? $c$ (length 1) is alive only at $t = 1$ ; $b$ (length 3) is alive at $t = 1, 2, 3$ ; $a$ (length 4) is alive at $t = 1, 2, 3, 4$ . So $t = 1$ has all three, $t = 2$ and $t = 3$ have ${a, b}$ , and $t = 4$ has only $a$ . $batch_sizes = [3, 2, 2, 1]$ (PyTorch stores this array 0-indexed internally, but the meaning of each entry is "alive count at the $t$ -th step" — we keep our $t = 1 \dots L_{m a x}$ labels.)
Check the sum. $\sum_{t} batch_sizes [t] = 3 + 2 + 2 + 1 = 8$ , and $\sum_{i} L_{i} = 4 + 3 + 1 = 8$ . Why this step? Every real token is stored exactly once; the two counts must agree — that is the invariant packing relies on.

Verify: $\sum$ batch_sizes $= 8 = \sum L_{i}$ . The packed data therefore has 8 rows, holding zero padding. ✓

Recall Why descending order?

Question: why must sequences be sorted longest-first before packing? ::: Because batch_sizes is non-increasing — at each step the still-alive sequences form a prefix of the rows, so the RNN processes rows $1 \dots batch_sizes [t]$ with no gaps.

Example 6 — C6: attention mask sends padding to exactly zero

Forecast: the padding score was the smallest anyway — does masking change the other two weights?

Softmax without masking (for contrast). With $exp (2) = 7.389$ , $exp (1) = 2.718$ , $exp (0.5) = 1.649$ , sum $= 11.756$ : $α^{no mask} \approx [0.6285, 0.2312, 0.1403]$ Why this step? Shows padding would wrongly steal $\approx 14%$ of the attention.
Mask: set $s_{3} = - \infty$ , so $exp (- \infty) = 0$ . New sum $= 7.389 + 2.718 + 0 = 10.107$ . Why this step? $- \infty$ before softmax is the standard trick to make a term vanish and renormalise the rest.
Masked weights: $α = [\frac{7.389}{10.107}, \frac{2.718}{10.107}, 0] \approx [0.7311, 0.2689, 0]$ Why this step? The real positions redistribute the padding's share among themselves — not just drop it.

Figure below (s03): a grouped blueprint bar chart, three key positions on the x-axis. Cyan bars = attention weights without the mask (padding still grabs ≈0.14); amber bars = weights after the $- \infty$ mask (padding is exactly 0, the two real bars grow taller). An amber arrow marks the padding bar collapsing to zero. It shows visually that the reclaimed mass moves to the real keys.

Verify: $0.7311 + 0.2689 + 0 = 1.0000$ (probability distribution) and $α_{3} = 0$ exactly. ✓ Note the real weights grew (0.6285→0.7311) because the padding mass was reclaimed.

Example 7 — C7: limiting case, the $d_{k}$ scale

Forecast: does the raw score's spread grow slowly or fast as we increase dimension?

Unscaled standard deviation $= d_{k}$ : $d_{k} = 1 \to 1, d_{k} = 64 \to 8, d_{k} = 512 \to 22.627$ Why this step? Large spread pushes softmax into saturation (one weight $\to 1$ , rest $\to 0$ ), killing gradients.
Scaled variance. $Var (q^{⊤} k / d_{k}) = \frac{Var ( q ^{⊤} k )}{( d _{k} ) ^{2}} = \frac{d _{k}}{d _{k}} = 1$ . Why this step? Dividing a random variable by constant $c$ divides its variance by $c^{2}$ ; choosing $c = d_{k}$ exactly cancels the $d_{k}$ growth.
Limiting behaviour. As $d_{k} \to \infty$ the unscaled spread $\to \infty$ (worse and worse), while the scaled spread stays pinned at $1$ — length- and dimension-agnostic.

Verify: $512 = 22.627 \dots$ and $512/ (512)^{2} = 512/512 = 1$ . ✓ across all three dimensions.

Example 8 — C8: real-world word problem (chatbot batch cost)

Here $L_{avg} = \frac{1}{B} \sum_{i} L_{i}$ is the average true length defined in the notation box, so $\sum_{i} L_{i} = B \cdot L_{avg}$ .

Forecast: one giant outlier message — guess whether the waste is "a little" or "brutal".

Total slots $= B \cdot L_{m a x} = 32 \cdot 128 = 4096$ . Why this step? Batch processing forces a rectangle at the longest row, so the model allocates work for every one of the $32 \times 128$ cells.
Real slots $= \sum_{i} L_{i} = B \cdot L_{avg} = 32 \cdot 12 = 384$ . Why this step? Real work is proportional to actual tokens, not padded width; and $\sum_{i} L_{i} = B L_{avg}$ follows straight from the definition of the average.
Waste fraction $= 1 - \frac{real}{total} = 1 - \frac{384}{4096} = 1 - 0.09375 = 0.90625 \approx 90.6%$ . Why this step? One outlier drags $L_{m a x}$ far above the average, so 90%+ of the grid is dead padding — this is the number that justifies bothering with packing.
Packing speedup $= \frac{total}{real} = \frac{4096}{384} \approx 10.67 \times$ fewer RNN steps. Why this step? An RNN's cost scales with the number of tokens processed; packing processes only the $384$ real ones, so the compute drops by the ratio of padded to real work.

Verify: $384/4096 = 0.09375$ , so waste $= 1 - 0.09375 = 0.90625$ ; speedup $4096/384 = 10.6 \overline{6}$ . Units: (slots)/(slots) is a dimensionless ratio, as a "fraction" and a "speedup factor" both must be. ✓ This is exactly why production systems either bucket by length (group similar-length messages so $L_{m a x} \approx L_{avg}$ ) or pack (skip the padding entirely).

Example 9 — C9: exam twist — the packing bug

Forecast: the numbers are fine — what property of the order is violated?

Diagnose. Classic pack_padded_sequence requires lengths sorted descending; $[2, 5, 3]$ is not. Why this step? Non-descending order breaks the "alive sequences are a prefix" invariant from Example 5, corrupting batch_sizes.
Sort descending and record where each row came from. Original rows are indexed $0, 1, 2$ with lengths $(2, 5, 3)$ . Ordering by length, largest first: $sorted lengths = [5, 3, 2] taken from original indices [1, 2, 0]$ Why this step? We must remember the source index of each moved row so we can undo the shuffle later; call this permutation $p = [1, 2, 0]$ ("sorted position $k$ came from original row $p [k]$ ").
Compute batch_sizes on the sorted lengths $[5, 3, 2]$ , using our 1-based columns $t = 1, \dots, 5$ (here $L_{m a x} = 5$ ). Alive count $=$ number of rows with $L_{i} \geq t$ : $t = 1 : 3, t = 2 : 3, t = 3 : 2, t = 4 : 1, t = 5 : 1 \Rightarrow batch_sizes = [3, 3, 2, 1, 1]$ Why this step? All 3 rows are alive through $t = 2$ ; the length-2 row dies after $t = 2$ (→ 2 at $t = 3$ ); the length-3 row dies after $t = 3$ (→ 1 at $t = 4, 5$ ). This is now non-increasing, so the invariant holds and packing succeeds.
Restore original order after the RNN. The RNN's outputs come back in sorted order; we need the inverse permutation $p^{- 1}$ so that original row $j$ is fetched from its sorted slot. Since $p = [1, 2, 0]$ , its inverse is $p^{- 1} = [2, 0, 1]$ (original row $0$ sits at sorted slot $2$ , original row $1$ at slot $0$ , original row $2$ at slot $1$ ). Why this step? Applying $p$ then $p^{- 1}$ returns the identity — the outputs land back in the caller's original ordering, unshuffled.

Verify: $\sum$ batch_sizes $= 3 + 3 + 2 + 1 + 1 = 10 = \sum L_{i} = 5 + 3 + 2$ . ✓ And applying permutation $p = [1, 2, 0]$ followed by its inverse $p^{- 1} = [2, 0, 1]$ returns the identity $[0, 1, 2]$ . ✓

Recall

Recall Test yourself

Why divide the masked loss by $\sum_{t} m_{i} (t)$ and not by $L_{m a x}$ ? ::: So each sequence is averaged over its real tokens; dividing by $L_{m a x}$ dilutes short sequences and biases the model toward them (Example 2: 2.0 vs wrong 1.5). In Example 8, what single feature of the batch caused 90% waste? ::: One outlier length-128 message pulling $L_{m a x}$ far above the average length 12. What does each entry of batch_sizes count? ::: The number of sequences still alive (real token present) at that time step; it must be non-increasing, which requires descending-length sorting. After masking with $- \infty$ , why did the real attention weights increase? ::: Softmax renormalises; the padding position's probability mass (≈14%) is redistributed to the real positions (Example 6: 0.6285→0.7311).

Related deep material: Transformers, Sequence-to-Sequence Models, LSTM and GRU.