Context window and sequence length: Visual walkthrough

Before any symbol appears, here is the vocabulary we will earn along the way:

Nothing else is assumed. Let's go.

Step 1 — Lay the tokens on a line

WHAT. Draw the input as a row of boxes, one per token. If the sentence is "the cat sat", that's three boxes. We label their positions $1, 2, 3, \dots, L$ .

WHY. Before we can count "comparisons," we need a clear picture of the things being compared. The row of boxes is that picture. The number written under the last box is exactly $L$ — this is the only quantity that will drive the whole derivation.

PICTURE. Below: five token-boxes in a line. The number $L$ at the end (in red) is the star of the whole page — it is the one knob you turn when you make the context window bigger.

Figure — Context window and sequence length

Step 2 — One token looks at all the others

WHAT. Pick a single box — say box $i$ (here $i$ just names which box we picked; think "box number $i$ "). To build its meaning, the model draws an arrow from box $i$ to every box on the line, including back to itself.

WHY. This is the defining rule of self-attention: a token's new representation is a blend of all tokens, so it needs a "how much do I care about you?" number pointing at each one. No box is allowed to be skipped — otherwise its information could never reach box $i$ . That "no skipping" is the seed of the quadratic cost.

PICTURE. One red box on the left fires arrows at all $L$ boxes. Count the arrows: exactly $L$ of them, because there are $L$ targets.

Step 3 — Every token does this — build the grid

WHAT. Step 2 was for one box. But every box must look at every other box. If we let box $i$ run over all $L$ rows and box $j$ (the target) run over all $L$ columns, the arrows fill a full square grid of cells. Cell $(i, j)$ = "how much box $i$ attends to box $j$ ."

WHY. We switched from "one row of arrows" to "a grid" because it lets us count everything at once. A grid with $L$ rows and $L$ columns has an obvious cell count, and counting cells is easier than counting tangled arrows.

PICTURE. An $L \times L$ square lights up. Row = who is looking (the "query"), column = who is being looked at (the "key"). One red cell marks a single pair $(i, j)$ so you can see that each cell is exactly one arrow from Step 2.

Step 4 — What lives inside one cell: the dot product

WHAT. Zoom into one cell $(i, j)$ . To get its number, the model takes the vector for box $i$ and the vector for box $j$ — each a list of $d$ numbers (call $d$ the model width, i.e. how long each token's number-list is) — and does a dot product: multiply matching entries, then add them all up.

WHY a dot product, and not something else? We need one single score that says "how aligned are these two tokens?" The dot product is the cheapest, most natural "alignment meter": when two lists point the same way, their products are large and the sum is big; when they clash, terms cancel and the sum is small. So it answers exactly the question a cell asks. It costs about $d$ multiply-add operations, because there are $d$ entries to multiply and add.

PICTURE. Two vertical stacks of $d$ numbers, matching entries joined by red multiply-links, all funnelling into one output number — the value of the cell.

Step 5 — Multiply: cost of the whole grid (COMPUTE)

WHAT. Now combine Step 3 (there are $L^{2}$ cells) with Step 4 (each cell costs $\sim d$ work). The total compute is cells times cost-per-cell.

WHY. "Compute" means "number of arithmetic operations the machine must actually perform." Total work is always (how many jobs) $\times$ (work per job). Here jobs $= L^{2}$ cells, work per job $= d$ .

PICTURE. The $L^{2}$ grid, with a little " $\times d$ " tag on the corner cell reminding you each cell hides a $d$ -long dot product. The whole thing is stamped $L^{2} \cdot d$ .

Step 6 — Memory is $L^{2}$ , not $L^{2} d$ — a crucial subtlety

WHAT. After each cell's dot product finishes, what gets stored is a single number (one scalar weight) per cell, so the softmax (see Self-Attention) can normalise them. The $d$ numbers that went into the dot product are not kept in the attention matrix — they collapse into that one output. So the stored grid is $L^{2}$ scalars.

WHY the distinction matters. It is the most common mistake around context cost (the parent note steel-mans it). Compute carries the $d$ because you must do $d$ multiplications per cell. Memory drops the $d$ because after the multiplication you keep only the sum — one number.

PICTURE. Left: the fat $d$ -deep dot product (Step 4) — that depth is compute. Right: the flat $L^{2}$ grid of single red dots — that flatness is memory. The depth vanishes once the sum is taken.

Step 7 — Edge cases: read the square at its extremes

A derivation is only trustworthy if it survives the corners. Let's push $L$ to its limits.

WHAT & WHY — the degenerate inputs:

$L = 0$ (empty input): a $0 \times 0$ grid has $0$ cells. Cost $= 0$ . The formula $L^{2} d = 0$ agrees — nothing to attend to, nothing to compute. ✅
$L = 1$ (single token): a $1 \times 1$ grid — one cell: the token attending to itself. Cost $\propto 1 \cdot d = d$ , essentially free. The quadratic hasn't "kicked in" yet because $1^{2} = 1$ . ✅
$L = n_{c t x}$ (the ceiling): the biggest legal square. This is the worst case the model was trained for; go one box further and there is no Positional Encoding slot for it — the picture would need a column with no address. ✅
Growth, not a jump: because area grows smoothly and steeply with side length, there is no sudden cliff in cost — but there is a sudden cliff in quality at $L > n_{c t x}$ , for the position reason above.

PICTURE. Three little squares in a row: a dot ( $L = 1$ ), a small grid ( $L = 4$ ), a big grid ( $L = 8$ ). The red highlights that going $4 \to 8$ (double the side) turns a $16$ -cell square into a $64$ -cell square — four times the area, not twice.

The one-picture summary

WHAT. One figure that fuses the whole chain: a line of $L$ tokens (Step 1) → arrows from one token (Step 2) → the full $L^{2}$ grid (Step 3) → the $d$ -deep dot product hiding in a cell (Step 4) → the two stamps $L^{2} d$ (compute) and $L^{2}$ (memory) (Steps 5–6).

Recall Feynman retelling — say it to a friend with no math

Imagine every word in a sentence has to shake hands with every other word to understand it. If there are $L$ words, the number of handshakes is a full square: $L$ people down the side, $L$ across the top, so $L \times L$ handshakes fill the grid — that's the $L^{2}$ . Each handshake isn't instant either: comparing two words means checking a list of $d$ little features and adding up the matches, so each handshake costs about $d$ units of effort. Multiply the handshakes by the effort each and you get $L^{2} \times d$ total work. But remembering the handshakes is cheaper: you only jot down a single "how firm was it?" number per handshake, so you store just $L^{2}$ numbers, no $d$ . And the punchline: if you double the number of people in the room, you don't double the handshakes — you quadruple them, because a square twice as wide has four times the area. That's the entire reason context windows can't just grow forever.

Recall Rebuild the whole page from four numbers

$L^{2}$ comes from where? ::: The area of the token-vs-token square — $L$ rows (who looks) times $L$ columns (who is looked at). Why does compute carry an extra $d$ but memory does not? ::: Each cell does a $d$ -long dot product ( $d$ multiply-adds → compute), but only stores the one number that comes out ( $1$ scalar → memory). What happens to cost when you double $L$ ? ::: It quadruples, because $(2 L)^{2} = 4 L^{2}$ . What is the smallest and largest legal square? ::: Smallest legal non-trivial is $1 \times 1$ ( $L = 1$ , one token attending to itself); largest is $n_{c t x} \times n_{c t x}$ , the trained ceiling.

Related: Self-Attention · Positional Encoding · Transformer Architecture · Truncation and Chunking · KV Cache · Tokenization