Context window and sequence length — Hinglish notes

4.2.10 · AI-ML › Tokenization & Language Modeling

YEH HAI KYA?

WHY distinction matter karti hai: sequence length tumhari choice hai per request; context window model ki hard ceiling hai. Inhe confuse karne se silent failures hoti hain jahan model lambi document ki shuruat "bhool" jaata hai.

LIMIT HAI KYO BILKUL?

Yeh limit arbitrary nahi hai — yeh self-attention cost se aati hai.

Cost ko scratch se derive karna

Step 1 — comparisons count karo. Length $L$ ki ek sequence ke liye, attention ek attention matrix $A$ banata hai jahan entry $A_{ij}$ = token $i$ kitna token $j$ ko attend karta hai.

Yeh step kyun? Kyunki attention sabhi positions pe ek weighted sum ki tarah define hota hai, isliye har (query, key) pair ka ek weight chahiye.

Entries ki sankhya hai: $# pairs = L \times L = L^{2}$

Step 2 — $A$ banane ki cost compute karo. Har entry dimension $d$ (model width) ke do vectors ka dot product hai. Ek dot product $\sim d$ multiply-adds cost karta hai. $L^{2}$ entries hain, isliye:

$Compute \propto L^{2} \cdot d$

Step 3 — memory. Softmax apply karne ke liye full attention matrix $A$ store karni padti hai. Iske $L^{2}$ entries hain (har ek ek single scalar weight hai), isliye: $Memory (attention weights) \propto L^{2}$ (Alag se, Q/K/V tensors sirf $O (L \cdot d)$ cost karti hain — $L$ mein linear. Dominant, quadratic term woh $L^{2}$ attention matrix hai.)

MODEL "POSITIONS" KAISE JAANTA HAI?

Agar attention sirf tokens pe sum kare, toh order kho jaata hai ("dog bites man" = "man bites dog"). Isliye hum positional information add karte hain.

WHY yeh purane models mein context cap karta hai: classic learned/sinusoidal position tables ek fixed length $n_{c t x}$ tak banaye gaye the. Position $n_{c t x} + 1$ maango aur uske liye simply koi encoding nahi → model sharply degrade ho jaata hai. Modern tricks (RoPE, ALiBi, interpolation) ise extend karte hain, lekin trained window hi woh jagah hai jahan quality guaranteed hai.

Figure — Context window and sequence length

Worked examples

Common mistakes (steel-manned)

Active recall

Recall Quick self-test (answers chhupao, pehle forecast karo)

Sequence length aur context window mein kya fark hai?
Attention compute $O (L^{2} d)$ kyun hai lekin attention memory $O (L^{2})$ ?
Agar $n_{c t x}$ 1k → 4k jaaye, kitna zyada attention compute?
Prompt kahan khatam hota hai aur output budget kahan shuru hota hai?

Answers: seq length = current input tokens (tumhari choice, $\leq$ ceiling); context window = trained max (fixed). Compute mein har pairwise dot product se extra $d$ aata hai; stored weights single scalars hain isliye memory sirf $L^{2}$ hai (plus Q/K/V ke liye $L d$ ). 16×. Dono same window share karte hain: $input + output \leq n_{c t x}$ .

Recall Feynman: ek 12-saal ke bachche ko samjhao

Socho tum ek story ek paper tube se padh rahe ho — ek baar mein sirf kuch words dikh sakte hain. Context window matlab tumhari tube kitni chaudi hai. Agar story tube se lambi hai, toh shuruat view se bahar khisak jaati hai aur tum usse bhool jaate ho. Zyada dekhne ke liye, tumhe chaudi tube chahiye — lekin tube ko do guna chauda karna tumhari aankhon ke liye chaar guna kaam hai, kyunki ab tumhe har word ko har doosre word se compare karna padta hai jo tum dekh sakte ho.

Flashcards

Ek language model ka context window kya hota hai?

Woh maximum number of tokens jo model ek baar mein process karne ke liye built/trained hua hai — ek fixed architectural ceiling (

n_{c t x}

Sequence length vs context window kya hai?

Sequence length = current input mein tokens (tumhari per-request choice); context window = woh fixed maximum jo model support karta hai. Requirement:

L \leq n_{c t x}

Self-attention compute

O (L^{2} \cdot d)

kyun hai?

Har token har doosre token ko attend karta hai (

L^{2}

query–key pairs) aur har pair ek

d

-dimensional dot product cost karta hai, jo

L^{2} d

deta hai.

Self-attention memory

O (L^{2})

kyun hai (

L^{2} d

nahi)?

Stored attention matrix har (query,key) pair ke liye ek scalar weight rakhti hai —

L^{2}

scalars.

d

factor sirf compute mein aata hai; Q/K/V storage alag

O (L d)

term hai.

Agar context length 2048 se 8192 jaaye, kitna zyada attention compute?

(8192/2048)^{2} = 4^{2} = 16 \times

Kya context window sirf prompt rakhta hai?

Nahi — prompt tokens AUR generated output tokens ek hi window share karte hain:

input + output \leq n_{c t x}

Kya context windows words mein measure hote hain?

Nahi, tokens (subword units) mein; English ~1.3 tokens/word hai, code/doosri languages ke liye zyada.

"Lost in the middle" effect kya hai?

Window ke andar bhi, models shuruat aur ant ki information ko beech ki information se kahin behtar use karte hain.

Purane models apni trained length ke baad kyun fail ho jaate hain?

Positional encodings sirf

n_{c t x}

tak exist karti hain; usse aage ki positions ka koi learned/defined encoding nahi hota, isliye quality collapse ho jaati hai.

6000-word English essay roughly kitne tokens ki hogi?

Lagbhag

6000 \times 1.3 \approx 7800

tokens.

Connections

Tokenization — window tokens mein count hoti hai, isliye tokenizer efficiency directly affect karti hai ki kitna text fit hoga.
Self-Attention — woh $O (L^{2})$ mechanism jo limit banata hai.
Positional Encoding — RoPE/ALiBi/interpolation decide karte hain ki window kitni extend ho sakti hai.
Transformer Architecture — jahan model width $d$ aur depth $L$ ke saath interact karte hain.
Truncation and Chunking — practical strategies jab $L > n_{c t x}$ ho.
KV Cache — stored keys/values ki memory bhi generation ke dauran sequence length ke saath badhti hai.