Visual walkthrough — Handling variable-length sequences

This page builds ONE central result from nothing: how do you train a network on a batch of sequences that have different lengths, so that the loss is fair to every sequence? The answer is the masked, length-normalised loss. We will draw it into existence, one step at a time.

Prerequisite ideas live in Handling variable-length sequences (the parent), and we lean lightly on Batch Processing, Masking in Deep Learning, and Recurrent Neural Networks.

---

Step 1 — The ragged batch: why nothing lines up

WHAT. We start with three sentences of different lengths. Think of each sentence as a row of boxes, one box per word.

WHY. Real data is ragged. Before we can talk about "loss", we have to see the problem: rows of unequal length cannot be stacked into a single grid, and a neural network layer only eats grids (tensors).

PICTURE. Below, three sentences: lengths $3$, $4$, and $1$. Notice the right edges do not align.

The subscript $i$ just names which sequence in the batch we mean. The batch has $B$ sequences; here $B=3$.

---

Step 2 — Padding: forcing a rectangle

WHAT. We find the longest sequence, call its length $L_{max}$, and glue extra fake tokens (value $0$, written <PAD>) onto every shorter sequence until all rows reach length $L_{max}$.

WHY. A batch must be a rectangle so the framework can multiply it as one tensor. $L_{max}=4$ here (the longest sentence), so every row is stretched to $4$ boxes.

PICTURE. The pale-yellow boxes are real; the pink hatched boxes are padding — placeholders with no meaning.

$$\text{padded}=\left[\begin{array}{cccc}1& 5& 7& 0\\ 2& 8& 3& 9\\ 4& 0& 0& 0\end{array}\right]$$

• Each row is one sequence.
• Each column is one time step $t\in \{1,2,3,4\}$.
• The $0$s are padding — they occupy space but must not influence learning.

---

Step 3 — The danger: padding leaks into the loss

WHAT. For every position the network outputs a prediction, and we score it with a per-token loss $\ell (y_i(t),{\hat{y}}_i(t))$.

WHY show this first. Before fixing the problem we must feel it. If we naively average the loss over all $B\times L_{max}$ cells, the fake padding cells contribute garbage. The network could "win" by learning to predict padding well — which is meaningless.

PICTURE. The red cells are padding cells whose loss values are pure noise, yet the naive average counts them.

We need a way to say: "only count $\ell$ where the token is real." That is exactly what a mask does.

---

Step 4 — The mask: a curtain of 1s and 0s

WHAT. We build a matrix the same shape as the padded batch. It holds $1$ over real tokens and $0$ over padding.

WHY this exact shape. A mask that lines up cell-for-cell with the loss grid lets us multiply them elementwise: $1\times \ell$ keeps a value, $0\times \ell$ erases it. Multiplication is the cheapest possible "keep / delete" switch.

PICTURE. The mask painted over the same grid: bright cells pass through, dark cells are shut off.

For our batch:

$$m=\left[\begin{array}{cccc}1& 1& 1& 0\\ 1& 1& 1& 1\\ 1& 0& 0& 0\end{array}\right]$$

Read it against the padded matrix in Step 2 — every $0$ token sits under a $0$ mask. The curtain is aligned.

---

Step 5 — Masking the loss: erase the padding terms

WHAT. Multiply each per-token loss by its mask, then sum across the row for sequence $i$: ${\sum }_{t=1}^{L_{max}}{m}_i(t)\cdot \ell (y_i(t),{\hat{y}}_i(t))$

WHY multiply-then-sum. Every padding term is multiplied by $0$ and vanishes; every real term survives untouched. So this sum equals only the loss over real tokens — the noise from Step 3 is gone.

PICTURE. Watch the pink padding terms collapse to zero while the yellow real terms pass through unchanged.

Term by term:

• $m_i(t)$ — the switch: $1$ keeps, $0$ deletes.
• $\ell (\cdots )$ — the penalty at that position.
• ${\sum }_{t=1}^{L_{max}}$ — we still loop over all columns (the rectangle), but the deleted ones add nothing.

We have a clean sum. But a raw sum is not yet fair — a longer sentence naturally accumulates more terms. That is Step 6.

---

Step 6 — Normalising: making long and short sentences fair

WHAT. Divide the masked sum by the number of real tokens, which is exactly ${\sum }_t{m}_i(t)=L_i$:

$${\mathcal{L}}_i=\frac{1}{\sum _{t=1}^{L_{max}}{m}_i(t)}\sum _{t=1}^{L_{max}}{m}_i(t)\cdot \ell (y_i(t),{\hat{y}}_i(t))$$

WHY divide by ${\sum }_t{m}_i(t)$. The sum of a mask row counts its 1s, which is the true length $L_i$. Dividing turns a total into an average per real token. Now a 4-word sentence and a 1-word sentence are judged on the same scale — average penalty per word — no matter how much padding was bolted on.

PICTURE. Left: raw sums (tall bars grow with length — unfair). Right: after dividing by length, the bars represent per-token error — comparable.

• ${\sum }_t{m}_i(t)=L_i$ — the denominator is the true length, computed from the mask itself (no separate bookkeeping needed).
• The whole fraction is the mean loss per real token for sequence $i$.

---

Step 7 — Averaging the batch

WHAT. Average the per-sequence losses over all $B$ sequences:

$${\mathcal{L}}_{batch}=\frac{1}{B}\sum _{i=1}^B{\mathcal{L}}_i$$

WHY. Batch Processing wants one scalar to backpropagate. Averaging (not summing) keeps the gradient magnitude independent of batch size, so your learning rate does not need re-tuning when $B$ changes.

PICTURE. Three fair per-sequence losses flow into one averaged batch loss.

• $B$ — number of sequences in the batch.
• ${\mathcal{L}}_i$ — the fair per-token loss from Step 6.
• ${\mathcal{L}}_{batch}$ — the single number that training minimises.

---

Step 8 — The degenerate cases (never let the reader hit one blind)

WHAT. We check the corners where the formula could break.

WHY. A formula you trust must survive its extremes. Each case below is a scenario a real batch can produce.

PICTURE. Three edge boards: an all-real sequence, a length-1 sequence, and the forbidden empty sequence.

• Case A — no padding needed ($L_i=L_{max}$). Every mask entry is $1$, denominator $=L_{max}$. The formula collapses to an ordinary average. Nothing special happens — good, it must reduce to the familiar case.
• Case B — a single-token sequence ($L_i=1$). Mask row is $[1,0,0,\dots ]$, denominator $=1$. The loss is just $\ell$ at that one token. Still well-defined.
• Case C — the empty sequence ($L_i=0$). Mask row is all zeros, denominator ${\sum }_t{m}_i(t)=0$ → division by zero. This is why frameworks (and you) must forbid empty sequences or add a tiny $\epsilon$ guard: $\frac{1}{\max ({\sum }_t{m}_i(t),\epsilon )}$. Never feed a length-0 sequence into this loss.

Recall Check yourself on the cases

If a sequence is fully real (no padding), what does the masked loss reduce to? ::: An ordinary mean over all $L_{max}$ tokens — the mask does nothing because every entry is 1. Why is a length-0 sequence dangerous here? ::: The mask sum is 0, so you divide by zero. Guard with an $\epsilon$ or drop empty sequences before batching.

---

The one-picture summary

Everything above, compressed: ragged sequences → pad to a rectangle → lay a mask curtain → multiply loss by mask (delete padding) → divide by mask-sum (fairness) → average over the batch.

$$\boxed{{\mathcal{L}}_{batch}=\frac{1}{B}\sum _{i=1}^B\frac{\sum _{t=1}^{L_{max}}{m}_i(t)\ell (y_i(t),{\hat{y}}_i(t))}{\sum _{t=1}^{L_{max}}{m}_i(t)}}$$

Recall Feynman retelling (say it out loud, no symbols)

"I have a stack of sentences of different lengths. To put them in one grid I glue fake words onto the short ones — that makes a rectangle the computer can chew. But those fake words are lies, so I lay a curtain over the grid: holes where the real words are, solid where the fakes are. I look at how wrong the model was in each box, and I only count the boxes I can see through the holes. Then — and this is the fair part — for each sentence I divide its total wrongness by how many real words it had, so a long sentence and a short one are graded on the same 'wrongness-per-word' scale. Finally I average those grades across all the sentences to get one number, and I nudge the network to make that number smaller. The only thing I must never do is feed a sentence with zero real words, because then I'd divide by zero — nothing to grade."

Related machinery worth a look next: Masking in Deep Learning applies this same curtain idea to attention and Transformers; LSTM and GRU and Sequence-to-Sequence Models consume these masked batches directly.