Foundations — Handling variable-length sequences

Before you can read the parent note, you must be able to read its alphabet. Below is every symbol and idea the parent uses, built from nothing, each one leaning on the ones before it.

1. What is a "sequence"?

A sequence is just an ordered list of things, one after another, where the order carries meaning.

• The picture: beads on a string. The first bead, the second bead, and so on. Swap two beads and you get a different string — order matters.
• Why the topic needs it: language, audio, and time series are all sequences. If order didn't matter we could just throw everything in a bag and use an ordinary network.

Look at the blue beads on the top row: they are labelled $x_1,x_2,x_3$. That subscript number is the position $t$.

2. The index $t$ and the length $L$

• The picture: $t$ is your finger pointing at one bead; $L$ is how many beads there are in total.
• Why needed: the parent writes things like $L_1,L_2,\dots ,L_B$ — those are the lengths of the 1st, 2nd, ... sequence. Different sequences have different $L$, and that difference is the whole problem.

3. Subscripts and the parenthesis notation $x_i(t)$

The parent mixes two ways of pointing at one element. Both mean the same thing.

• The picture: a grid. Rows are different sequences; columns are positions. $x_i(t)$ is the cell in row $i$, column $t$.
• Why needed: we process many sequences at once, so we always need two addresses — which sentence, and which word in it.

4. A "batch" and the letter $B$

• The picture: a stack of index cards, each card one sentence. $B$ = number of cards in the stack.
• Why needed: hardware (GPUs) is fast when it does the same operation to many items at once. The parent's ${\mathcal{L}}_{batch}=\frac{1}{B}{\sum }_{i=1}^B{\mathcal{L}}_i$ just says "average the loss over all $B$ cards in the stack". (See Batch Processing.)

5. The tensor and why it must be rectangular

• The picture: a brick wall where every row must be the same width — no jagged edges allowed.
• Why needed: this is the villain of the whole topic. Sequences have different lengths (jagged), but a tensor demands equal lengths (rectangular). That clash is exactly what padding fixes.

The left picture (jagged) is what real data looks like. The right picture (rectangular) is what the computer demands. The grey cells are the fix — see the next section.

6. Padding and the <PAD> token

• The picture: the grey cells on the right of figure s02 — extra beads glued on to make all strings equal length.
• Why needed: it turns the jagged wall into a rectangular one. Now the batch is a legal tensor.

• The picture: the width of the widest row; every other row stretches out to match it.

7. The mask $m_i(t)$ — the star of the show

• The picture: a stencil laid over the batch. Holes (1s) let real data through; blocked spots (0s) hide the fake padding.
• Why needed: whenever we compute something (a loss, an attention score), we multiply by the mask so padding contributes zero. It's the memory of "which parts were fake".

Read $m_i(t)$ from the picture: row $i$ is all 1s up to its true length $L_i$, then all 0s. The green region is real, the dark region is padding.

8. The Greek letter $\Sigma$ (sum) and ${\sum }_{t=1}^{L_{max}}$

• The picture: a conveyor belt. $t$ starts at 1, steps to 2, 3, ... up to $L_{max}$; at each stop you drop the value $a_t$ into a total.
• Why needed: the parent's masked loss sums over all positions. Multiplying by $m_i(t)$ inside the sum means the padding positions add $0$, so only real positions count.

9. Loss $\ell$ and $\mathcal{L}$

• Two fonts, two meanings (the parent uses both):
  • $\ell (y_i(t),{\hat{y}}_i(t))$ — the per-position loss: how wrong the prediction was at one single position $t$.
  • ${\mathcal{L}}_i$ (fancy script L) — the whole-sequence loss for sequence $i$: an average of the per-position losses over its real tokens.
• The picture: $\ell$ is one student's score on one question; ${\mathcal{L}}_i$ is that student's average over all real questions (skipping the blank padding questions).

10. The hidden state $h_t$ and the RNN

• The picture: reading a book and keeping a mental summary that you update after each word. $h_t$ = your summary after word $t$; $h_{t-1}$ = your summary just before it.
• Why needed: a sequence model must remember earlier elements to understand later ones. The parent's update $h_t=\tanh (W_{hh}{h}_{t-1}+W_{xh}{x}_t+b_h)$ says "new memory = blend of (old memory) and (new input)". See Recurrent Neural Networks and LSTM and GRU.

11. Vectors, dot product, and ${\mathbb{R}}^d$

The attention section leans on these.

• The picture: an arrow pointing somewhere in space; $d$ is how many directions (axes) that space has. $q\in {\mathbb{R}}^{d_k}$ means "$q$ is an arrow with $d_k$ numbers".

• The picture: two flashlight beams. Pointing the same way → strong overlap (large dot product); at right angles → no overlap (zero).
• Why needed: attention uses $q^T{k}_j$ to ask "how well does what I'm looking for (query) match what's here (key)?" See Attention Mechanism and Transformers.

12. Softmax and $\alpha$

• $\exp$ means $e^{(\cdot )}$: it makes everything positive and blows up big scores more than small ones (winner emphasised).
• ${\alpha }_j$ (Greek alpha) is the resulting weight for position $j$ — "how much attention position $j$ gets", between 0 and 1.
• Why needed: attention's output is ${\sum }_j{\alpha }_j{v}_j$, a weighted blend of values. Softmax guarantees the weights are a fair share adding to 100%.

How it all fits together

Everything on the left builds the padding + mask machinery; the middle builds the RNN; the right builds attention. All three converge on the parent topic: Handling variable-length sequences.

Equipment checklist

Test yourself — reveal only after you've answered.