Foundations — Context window and sequence length

Before you can understand "context window and sequence length", you must be able to read every symbol on that page without flinching. This page builds each one from nothing — plain words first, then a picture, then why the topic needs it. Read top to bottom; each brick rests on the one below it.

0. The very first object: a token

Everything starts here. A model does not read letters or words — it reads tokens.

The picture: imagine a long strip of paper with your sentence written on it, and scissors that snap it into little labelled cards. Each card is a token.

Why the topic needs it: the whole page counts things in tokens, never in words or letters. "internationalization" is one word but five tokens. If you measure the wrong unit, every budget calculation on the parent page is wrong. See Tokenization for how the cutting is decided.

1. The symbol $L$ (also written $T$) — sequence length

Now that we can count tokens, we can count how many are in front of the model right now.

The picture: lay your token-cards in a row and count them left to right: card 1, card 2, ... card $L$. The last card's number is $L$.

Why the topic needs it: $L$ is the star of every cost formula. When the parent page writes "$L^2$" it means "(this many tokens) squared". You cannot read the topic without owning $L$.

2. The ceiling $n_{ctx}$ — context window

$L$ can grow... but not forever. There is a wall.

The picture: think of a paper tube (a telescope) held over the row of token-cards. The tube has a fixed width — that width, measured in cards, is $n_{ctx}$. Any card outside the tube is invisible.

Why the topic needs it: the entire page is the story of this inequality — what happens when your text is too long, why the wall exists, and what it costs to move the wall. $L$ is the variable; $n_{ctx}$ is the constant.

3. The engine that forces the wall: self-attention

Why is there a wall at all? Because of how the model looks. That mechanism is self-attention (full detail in Self-Attention). We only need its shape here.

The picture: draw a dot for each token, then draw an arrow from every dot to every other dot. That grid of arrows is the source of all the cost.

Why the topic needs it: this "everyone-with-everyone" habit is exactly why doubling the window quadruples the work. Hold this picture — the next symbols just count its arrows.

4. The attention matrix $A$ and its entries $A_{ij}$

We need a place to write down "how strongly does $i$ look at $j$?" for every pair. That place is a grid — a matrix.

The picture: an $L\times L$ chessboard. Row = the token doing the looking, column = the token being looked at. Fill each square with one weight.

Why the topic needs it: the parent page says the model "must store the attention matrix". Now you can see it: an $L$-by-$L$ board of squares. Counting those squares gives the famous $L^2$.

5. The symbol $d$ — model width (embedding dimension)

Each token is not a single number; it is a list of numbers — a small vector. How long is that list? That length is $d$.

The picture: replace each token-card with a thin column of $d$ stacked numbers. Two tokens are "similar" if their columns point the same way.

Why the topic needs it: filling one square of the attention board means comparing two of these $d$-long columns. That comparison is the dot product, our next symbol — and it costs about $d$ steps, which is where the extra $d$ in the compute formula comes from.

6. The dot product — why one comparison costs $\sim d$

We said "compare two columns". Which comparison, and why does it cost $d$?

The picture: two arrows in space. Their dot product is large when they lean the same direction, zero when perpendicular. That single number becomes one square $A_{ij}$ on the board.

Why this tool and not another? We need one number saying "how relevant is token $j$ to token $i$?" The dot product is the cheapest, standard measure of alignment — and because it does $d$ multiply-adds, each square costs about $d$ operations.

7. The symbol $\mathcal{O}(\cdot )$ — big-O, "grows like"

The parent page writes $\mathcal{O}(L^{2}d)$. What does that curly $\mathcal{O}$ mean?

The picture: a graph. $\mathcal{O}(L)$ is a straight line; $\mathcal{O}(L^2)$ is an upward-curving parabola. The parabola crushes the line as $L$ grows — that curve is the reason context windows are finite.

Why the topic needs it: it lets us say "doubling $L$ quadruples the work" in one clean symbol, without pretending we know the exact machine cost.

8. The symbol $\text{pos}$ — position, and why order needs saving

Attention sums over tokens, and a sum does not care about order ($3+5=5+3$). So "dog bites man" would look the same as "man bites dog". We must inject where each token sits.

The picture: number every token-card 0,1,2,... and stamp that number onto its column of $d$ values before attention runs.

Why the topic needs it: older models built a position table only up to $n_{ctx}$ seats. Ask for seat $n_{ctx}+1$ and there is no stamp for it — a second, independent reason the window has a wall (the first being the $L^2$ cost).

How the foundations feed the topic

Read the map bottom-up: tokens give $L$; the everyone-vs-everyone rule turns $L$ into an $L\times L$ board; counting squares gives $L^2$, and the per-square dot product adds the $d$; big-O names the shapes; position limits and the quadratic cost together build the wall that is the context window. All of this sits under Transformer Architecture.

Equipment checklist

Hide the answers; forecast each before revealing.