Context window and sequence length: Exercises

This page is a self-testing ladder. Each rung is harder than the last:

L1 Recognition — can you spot / recall the right idea?
L2 Application — plug numbers into one rule.
L3 Analysis — combine rules, reason about why.
L4 Synthesis — design / trade off across several ideas.
L5 Mastery — the kind of question an engineer actually gets paid to answer.

Every problem hides its worked solution inside a collapsible [!recall]- callout — forecast your answer first, then open it. After each level there is a [!mistake] callout for the exact trap most people fall into at that level.

Prerequisites you may want open: the parent topic, plus Tokenization, Self-Attention, Positional Encoding, Transformer Architecture, Truncation and Chunking, and KV Cache.

Level 1 — Recognition

L1.1

A model card says: "Context length: 8192." You send it a 500-token question (and it has not generated anything yet). What is $L$ ? What is $n_{c t x}$ ? Is the request valid?

Recall Solution

$L = 500$ (the tokens currently in the window — here just your input, since nothing is generated yet).
$n_{c t x} = 8192$ (the fixed trained ceiling).
Valid, because $L \leq n_{c t x}$ : $500 \leq 8192$ . ✅

What we did: matched each number to its symbol. The card number is always the ceiling $n_{c t x}$ ; the current window occupancy is $L$ .

L1.2

True or false: "If my text fits inside the context window, the model uses every token equally well."

Recall Solution

False. Fitting only guarantees the tokens are visible. The "lost in the middle" effect means tokens at the start and end are attended to far better than tokens buried in the middle. Fitting ≠ attending well.

L1.3

Which quantity is your per-request choice, and which is fixed by the model: sequence length or context window?

Recall Solution

Sequence length $L$ — under your control (you decide how long the prompt is, and how many tokens you ask the model to generate).
Context window $n_{c t x}$ — fixed by the trained weights/architecture; you cannot raise it by asking.

Level 2 — Application

L2.1

An essay is 6000 words. Using $1.3$ tokens/word, estimate $L$ . If $n_{c t x} = 4096$ , does it fit?

Recall Solution

$L \approx 6000 \times 1.3 = 7800 tokens$ Why multiply by 1.3? subword tokenizers cut rare/long words into several tokens, so tokens outnumber words. Since $7800 > 4096$ , it does not fit — you must truncate or chunk.

L2.2

Model A has $n_{c t x} = 2048$ ; Model B has $n_{c t x} = 8192$ . By what factor does B's window exceed A's, and by what factor does its attention compute exceed A's?

Recall Solution

Window factor: $8192/2048 = 4 \times$ . Attention compute $\propto L^{2}$ , so: $\frac{819 2 ^{2}}{204 8 ^{2}} = 4^{2} = 16 \times$ Why squared? every token attends to every token ⇒ $L \times L$ pairs. A $4 \times$ window is $16 \times$ the attention work.

L2.3

A chat model has $n_{c t x} = 8192$ . Your system prompt + user message = 6000 tokens. How many tokens remain for the model's reply?

Recall Solution

$output budget = 8192 - 6000 = 2192 tokens$ Why subtract? the window holds prompt + generated output together — they draw from one shared budget, and as generation proceeds $L$ climbs from 6000 toward the ceiling. Ask for a 3000-token reply and it gets cut off at 2192.

Level 3 — Analysis

L3.1

Explain in one line each why attention compute is $O (L^{2} d)$ but attention memory (the stored weights) is only $O (L^{2})$ .

Recall Solution

Compute $O (L^{2} d)$ : there are $L^{2}$ query–key pairs, and each pair's score is a dot product of two $d$ -dimensional vectors, costing $\sim d$ multiply-adds. Multiply: $L^{2} \times d$ .
Memory $O (L^{2})$ : once computed, each pair collapses to one scalar weight. Storing $L^{2}$ scalars is $L^{2}$ , with no $d$ . (Q/K/V tensors add a separate $O (L d)$ .) Key idea: the $d$ lives inside the dot product (compute), not in the stored result (memory).

L3.2

Two setups cost the same attention compute. Setup X: $L = 4000$ , $d = 512$ . Setup Y: $L = 2000$ , $d = ?$ . Find $d$ for Y so that $L^{2} d$ matches X.

Recall Solution

Set the products equal: $400 0^{2} \times 512 = 200 0^{2} \times d$ $d = \frac{400 0 ^{2} \times 512}{200 0 ^{2}} = (\frac{4000}{2000})^{2} \times 512 = 4 \times 512 = 2048$ What this shows: halving $L$ cuts $L^{2}$ by $4 \times$ , so $d$ must grow $4 \times$ to keep the same compute. Sequence length dominates because it enters squared; width $d$ enters only linearly.

L3.3

The parent note warns older models "degrade sharply" beyond position $n_{c t x}$ . Using Positional Encoding, explain the mechanism — what specifically is missing at position $n_{c t x} + 1$ ?

Recall Solution

Classic learned/sinusoidal position tables are precomputed for positions $0 \dots n_{c t x} - 1$ . At position $n_{c t x}$ (index $n_{c t x} + 1$ in 1-based counting) there is literally no entry / an untrained region in that table.

Without a position signal, Self-Attention cannot tell order apart ("dog bites man" = "man bites dog").
So the model must either wrap, extrapolate, or output garbage — hence the sharp cliff. Modern fix: RoPE / ALiBi / interpolation let positions extend smoothly, but quality is only guaranteed inside the trained window.

Level 4 — Synthesis

L4.1

You must fit a document of 30,000 tokens through a model with $n_{c t x} = 8192$ , reserving 2000 tokens in each call for a fixed instruction + output. Design a chunking plan: how many tokens of document per chunk, and how many chunks minimum?

Recall Solution

Step 1 — usable room per call. Overhead is 2000, so document room per chunk: $8192 - 2000 = 6192 tokens$ Step 2 — number of chunks. Divide and round up (a partial chunk still needs its own call): $⌈ \frac{30000}{6192} ⌉ = ⌈ 4.844 \dots ⌉ = 5 chunks$ Why round up? $4$ chunks cover only $4 \times 6192 = 24768 < 30000$ ; the leftover $5232$ tokens need a 5th call.

The figure below shows all 5 calls stacked. In each row the plum block is the fixed 2000-token overhead (instruction + reserved output), the teal block is the document tokens carried in that call (6192 for calls 1–4, and only 5232 for call 5), and the dotted region is the unused room up to the orange ceiling $n_{c t x} = 8192$ . Reading down the teal blocks, $4 \times 6192 + 5232 = 30000$ confirms the whole document is covered.

Figure — Context window and sequence length

L4.2

For that same plan, compare total attention compute of the 5-chunk approach (each chunk runs at its own length) vs a hypothetical single model that could swallow all 30,000 tokens at once. Assume same $d$ ; use $L^{2}$ as the compute proxy.

Recall Solution

Single giant call: $L = 30000$ , compute proxy $= 3000 0^{2} = 9.0 \times 1 0^{8}$ . Chunked: four full chunks at $L = 6192$ plus one at $L = 30000 - 4 \times 6192 = 5232$ : $4 \times 619 2^{2} + 523 2^{2} = 4 (38, 340, 864) + 27, 373, 824 = 153, 363, 456 + 27, 373, 824 = 1.807 \times 1 0^{8}$ Ratio: $\frac{9.0 \times 1 0 ^{8}}{1.807 \times 1 0 ^{8}} \approx 4.98 \times$ . Interpretation: chunking is ~5× cheaper in attention here — because the $L^{2}$ law punishes one huge sequence far more than several medium ones. The trade-off: chunks lose cross-chunk attention, so the model can't relate token 1 to token 29,000.

Level 5 — Mastery

L5.1

A production API charges by total tokens processed and separately warns that attention compute scales as $L^{2}$ . You have a fixed 100,000-token corpus to answer questions over. Argue quantitatively why "just use one huge 128k-context model per query" can be worse than retrieval + a small window — and identify the one place where the big window wins.

Recall Solution

Setup. Say the small approach retrieves only the ~ $4000$ relevant tokens; the big approach feeds all $100000$ . Attention-compute proxy ( $L^{2}$ ): $big : 10000 0^{2} = 1.0 \times 1 0^{10}, small : 400 0^{2} = 1.6 \times 1 0^{7}$ $ratio = \frac{1.0 \times 1 0 ^{10}}{1.6 \times 1 0 ^{7}} = 625 \times$ The big window is 625× more attention compute for the same question — and pays for it every query, forever. Where the big window wins: when the answer requires global reasoning across the whole corpus (e.g. "summarize contradictions between page 3 and page 290"). Retrieval that pulls only 4000 tokens may never co-locate the two facts that must be compared, so it can be wrong at any price. Big context buys cross-document attention that chunking/retrieval structurally cannot. The trade: cost ( $L^{2}$ ) vs completeness (global attention). Neither is free.

L5.2

Derive the general rule: for a fixed overhead $h$ tokens per call and window $n_{c t x}$ , the minimum number of calls to cover a document of $D$ tokens is $N = ⌈ \frac{D}{n _{c t x} - h} ⌉ .$ Then explain what happens as $h \to n_{c t x}$ , and verify with $D = 30000, n_{c t x} = 8192, h = 2000$ .

Recall Solution

Derivation. Each call has $n_{c t x} - h$ tokens of usable room for the document (the rest is overhead). To cover $D$ tokens you need enough calls so that (calls) $\times$ (room) $\geq D$ : $N \times (n_{c t x} - h) \geq D ⟹ N \geq \frac{D}{n _{c t x} - h} ⟹ N = ⌈ \frac{D}{n _{c t x} - h} ⌉ .$ The ceiling appears because a fraction of a call still costs a whole call. Degenerate limit $h \to n_{c t x}$ : the denominator $n_{c t x} - h \to 0^{+}$ , so $N \to \infty$ — if overhead eats the whole window there is no room left for the document; the task is impossible. (And $h > n_{c t x}$ makes room negative → undefined, meaning the instruction alone already overflows.) Verify: $⌈ \frac{30000}{8192 - 2000} ⌉ = ⌈ \frac{30000}{6192} ⌉ = ⌈ 4.844 ⌉ = 5$ . ✅ Matches L4.1.

Wrap-up recall

Recall One-line answers to the whole ladder

L1: $L$ = tokens currently in the window (prompt + generated so far); $n_{c t x}$ = fixed trained max; $L \leq n_{c t x}$ .
L2: 7800 tokens (no fit); $16 \times$ compute; 2192 output tokens.
L3: compute $O (L^{2} d)$ (dot products), memory $O (L^{2})$ (scalars); $d = 2048$ ; missing position table entry.
L4: 6192 doc-tokens/chunk, 5 chunks; chunking ~5× cheaper but loses cross-chunk attention.
L5: big window $625 \times$ compute but wins on global reasoning; $N = ⌈ D / (n_{c t x} - h)⌉ = 5$ .