WHY the distinction matters: sequence length is your choice per request; context window is the model's hard ceiling. Confusing them causes silent failures where the model "forgets" the start of a long document.
Step 1 — count the comparisons. For a sequence of length L, attention builds an attention matrixA where entry Aij = how much token i attends to token j.
Why this step? Because attention is defined as a weighted sum over all positions, so we need a weight for every (query, key) pair.
The number of entries is:
#pairs=L×L=L2
Step 2 — compute cost of building A. Each entry is a dot product of two vectors of dimension d (the model width). One dot product costs ∼d multiply-adds. There are L2 entries, so:
Compute∝L2⋅d
Step 3 — memory. The full attention matrix A must be stored to apply softmax. It has L2 entries (each a single scalar weight), so:
Memory (attention weights)∝L2(Separately, the Q/K/V tensors cost only O(L⋅d) — linear in L. The dominant, quadratic term is the L2 attention matrix.)
If attention just sums over tokens, order is lost ("dog bites man" = "man bites dog"). So we add positional information.
WHY this caps context in older models: classic learned/sinusoidal position tables were built up to a fixed length nctx. Ask for position nctx+1 and there is simply no encoding for it → the model degrades sharply. Modern tricks (RoPE, ALiBi, interpolation) extend this, but the trained window is still where quality is guaranteed.
What is the difference between sequence length and context window?
Why is attention computeO(L2d) but attention memoryO(L2)?
If nctx goes 1k → 4k, how much more attention compute?
Where does prompt end and output budget begin?
Answers: seq length = current input tokens (your choice, ≤ ceiling); context window = trained max (fixed). Compute has an extra d from each pairwise dot product; the stored weights are single scalars so memory is just L2 (plus Ld for Q/K/V). 16×. They share the same window: input+output≤nctx.
Recall Feynman: explain it to a 12-year-old
Imagine reading a story through a paper tube — you can only see a few words at a time. The context window is how wide your tube is. If the story is longer than the tube, the beginning slides out of view and you forget it. To see more, you'd need a wider tube — but making the tube twice as wide is four times as much work for your eyes, because you now have to compare every word with every other word you can see.
The maximum number of tokens the model was built/trained to process at once — a fixed architectural ceiling (nctx).
What is sequence length vs context window?
Sequence length = tokens in the current input (your per-request choice); context window = the fixed maximum the model supports. Requirement: L≤nctx.
Why is self-attention computeO(L2⋅d)?
Each token attends to every other token (L2 query–key pairs) and each pair costs a d-dimensional dot product, giving L2d.
Why is self-attention memoryO(L2) (not L2d)?
The stored attention matrix holds one scalar weight per (query,key) pair — L2 scalars. The d factor only appears in compute; Q/K/V storage is a separate O(Ld) term.
If context length goes from 2048 to 8192, how much more attention compute?
(8192/2048)2=42=16×.
Does the context window hold only the prompt?
No — prompt tokens AND generated output tokens share the same window: input+output≤nctx.
Are context windows measured in words?
No, in tokens (subword units); English is ~1.3 tokens/word, more for code/other languages.
What is the "lost in the middle" effect?
Even within the window, models use information at the start and end far better than in the middle.
Why do older models fail past their trained length?
Positional encodings only exist up to nctx; positions beyond it have no learned/defined encoding, so quality collapses.
Roughly how many tokens is a 6000-word English essay?
Socho ek Transformer model ek lambe scroll ko ek paper tube ke through padh raha hai. Jitne tokens us tube ke andar aa sakte hain, wahi context window hai — model ka fixed maximum (nctx). Aur sequence length (L) matlab abhi aapne jo actual input diya hai uske tokens. Rule simple hai: L≤nctx. Agar aapka document context window se bada ho gaya, toh shuru ka hissa tube se bahar slide ho jayega aur model use "bhool" jayega. Yaad rakho — ye sab tokens me counted hota hai, words me nahi. English me roughly 1.3 tokens per word lagao.
Ab sawaal — ye limit hai hi kyun? Iska reason hai self-attention. Har token baaki sabhi tokens ko dekhta hai, toh L×L=L2 pairs ban-te hain. Isliye computeO(L2⋅d) hai (har pair ek d-dimensional dot product) aur memoryO(L2) (attention matrix me har pair ka bas ek scalar weight store hota hai; d wala factor sirf compute me aata hai). Q/K/V ka storage alag se O(L⋅d) hota hai — linear. Matlab agar context 2× kar do, cost 4× ho jaati hai. Isliye long-context models mehnge hote hain — ye koi arbitrary decision nahi, quadratic maths hai.
Ek practical baat: context window sirf input ke liye nahi hota — prompt + generated output dono usi window me fit hote hain. Agar prompt 6000 tokens le liya aur nctx=8192 hai, toh output ke liye sirf 2192 tokens bache. Aur ek trap: bade window me bhi model "lost in the middle" ho jaata hai — beech ka content theek se use nahi hota. Isliye important instructions ko shuru ya end me rakho.
Exam aur real projects dono me ye samajhna zaroori hai: token budgeting galat kiya toh model chup-chaap document ka aadha hissa kaat dega aur aapko galat ya adhoora answer milega. Toh hamesha tokens estimate karo, L2 cost yaad rakho, aur window ko input+output dono ke liye plan karo.
