KV-cache optimization: Question bank — Hinglish deep dive

6.1.13 · D5 · AI-ML › Scaling & Efficient Architectures › KV-cache optimization

Definition Is page mein use hone wale symbols (pehle ye padho)

Neeche ki har cheez kuch gireh letters se bani hai. Yahan har ek ka plain words mein matlab diya gaya hai, use karne se pehle. Dimag mein figure 1 ki past tokens ki strip ki tasveer rakho.

Token — text ka ek chunk (ek word ya word-piece). Model inhe ek-ek karke, left to right generate karta hai.
$L$ — ab tak sequence mein tokens ki sankhya (uski length).
$x_{t}$ — numbers ka vector ("embedding") jo position $t$ par baitha token represent karta hai.
$Q$ (query), $K$ (key), $V$ (value) — teen vectors jo model har token se banata hai. Socho ek token library mein ja raha hai: uski query hai "main kya dhundh raha hoon?", uski key hai "ye mera label hai taaki doosre mujhe dhundh sakein", uski value hai "ye wo content hai jo main de dunga agar koi mujhe choose kare". Cache sirf $K$ aur $V$ store karta hai.
$W_{Q}, W_{K}, W_{V}$ — teen fixed weight matrices (numbers ki grids) jo training ke dauran seekhi gayi hain. Kisi token ki query/key/value paane ke liye uski embedding ko matching matrix se multiply karo: $Q_{t} = W_{Q} x_{t}$ , $K_{t} = W_{K} x_{t}$ , $V_{t} = W_{V} x_{t}$ . Kyunki training khatam hone ke baad $W$ matrices frozen hoti hain, kisi fixed token ke liye $K_{t}$ aur $V_{t}$ kabhi nahi badlte — yahi fixedness caching ke kaam aane ki poori wajah hai.
$d$ — model dimension: har token ke embedding vector mein kitne numbers hain.
head — attention kai parallel copies mein split hota hai jise heads kehte hain, har ek vector ke alag slice ko dekhta hai. $h$ = heads ki sankhya.
$d_{k}$ — head dimension: har single head kitne numbers ke saath kaam karta hai (toh $d \approx h \cdot d_{k}$ ).

Kaam karte waqt ye box khula rakho — neeche ke zyaadatar traps asal mein in mein se kaunsa har step par badalta hai aur kaunsa fixed rehta hai ke baare mein hain.

Upar figure 1 dekho: baayein taraf ke fixed grey blocks cached $K, V$ hain past tokens ke; daayein taraf ka bright block bilkul naya token hai, jise ek fresh query chahiye lekin sirf ek key aur ek value append karta hai.

True or false — justify

T/F jawab do, phir reason bhi do. Sirf "True" likhne se zero milega — reasoning hi point hai.

KV-cache, no-cache ki tulna mein generation ka mathematical output change kar deta hai.

False. Cache ek pure memoisation trick hai — ye wahi numbers store karta hai unhe recompute karne ki jagah. Output bit-identical hota hai (floating-point non-determinism tak); sirf speed aur memory differ karti hai.

KV-caching inference ke dauran model ki total memory use ko reduce karta hai.

False. Ye memory badhata hai: ab aap har past token ke liye

K, V

store kar rahe ho. Ye extra memory ko saved compute ke badle trade karta hai — memory saving ka ulta.

KV-caching training ke saath-saath generation mein bhi speedup deta hai.

False. Training teacher forcing use karti hai — saare

L

tokens ek saath present hote hain, isliye har

K_{i}, V_{i}

exactly ek baar compute hota haiैसे भी. Steps ke across kuch reuse karne ke liye nahi hai, isliye cache sirf autoregressive generation mein help karta hai.

Token 5 ki keys aur values constant rehti hain jab hum tokens 6, 7, 8, … generate karte hain.

True. Weight matrices

W_{K}, W_{V}

training ke baad frozen hain aur token 5 ki embedding

x_{5}

fixed hai, isliye

K_{5} = W_{K} x_{5}

aur

V_{5} = W_{V} x_{5}

kabhi nahi badlte. Yahi fixedness exactly kyun hai ki wo cacheable hain.

Multi-Query Attention cache size utna hi reduce karta hai jitna model parameters reduce karta hai.

False. MQA cache ko

h

(heads ki sankhya) ke factor se shrink karta hai, lekin parameter saving bahut choti hai — sirf

K, V

projection matrices shrink hoti hain, aur wo poore model ka ek chota slice hain. Fayda inference par memory-bandwidth ka hai, parameter count ka nahi.

KV-cache ke saath, per-token generation cost pehle kitne tokens aaye usse independent ho jaati hai.

Mostly false. Projection cost per token ab constant hai (

O (d^{2})

, jahan

d

model dimension hai), lekin attention score step abhi bhi

O (t \cdot d)

cost karta hai kyunki naya query

Q_{t}

ko saare

t

cached keys ke saath dot karna padta hai. Latency context length ke saath abhi bhi badhti hai, bas overall quadratic ki jagah linearly.

Size

W

ka sliding-window cache memory ko sequence length se independent constant banata hai.

True. Aap sirf last

W

keys/values rakhte ho, isliye cache memory

O (W)

par cap ho jaati hai. Cost ye hai ki

W

steps se pehle ke tokens literally invisible hain — long-range dependencies drop ho jaati hain.

Grouped-Query Attention ek special case hai jisme MQA aur standard attention dono endpoints ke roop mein hain.

True. GQA

h

query heads ko

g

KV-groups mein group karta hai.

g = h

set karne se standard multi-head attention wapas aata hai;

g = 1

set karne se MQA wapas aata hai. GQA inke beech interpolate karta hai.

Spot the error

Har line ek plausible-sounding claim hai jisme ek hidden flaw hai. Flaw batao.

Q

ko bhi cache karein — isme bhi matrix multiply chahiye, toh aur bhi compute bachega."

Flaw ye hai: har naya token ek fresh query

Q_{t}

generate karta hai jo kabhi reuse nahi hoti, kyunki query encode karti hai "yeh naya token kya dhundh raha hai". Sirf

K, V

(jo past tokens offer karte hain) reuse hote hain.

Q

ko cache karna kuch nahi bachata.

"Kyunki cache mein pehle se har key hai, token

t

unhe freely attend kar sakta hai."

Causal mask missing hai. Token

t

ko sirf positions

\leq t

dekhne chahiye. Agar future keys kabhi cache mein baith jaayein (jaise kuch batched/padded setups mein), unhe attend karna woh information leak kar deta hai jo model ko nahi honi chahiye.

"Heads

h

ki sankhya double karne se compute double hoti hai lekin cache size same rehti hai."

Standard attention ke liye galat: cache size

2 L h d_{k} \times

precision hai (jahan

h

= heads,

d_{k}

= head dimension), isliye ye $h$ ke saath linearly scale karta hai। Heads double karne se cache bhi double ho jaata hai. (Yahi exactly kyun MQA/GQA exist karte hain.)

"MQA aur GQA lossless hain — ye sirf same numbers ko reorganise karte hain."

False. Query heads ke across

K, V

share karne se representational capacity hat jaati hai, isliye model ko iske liye train ya fine-tune karna padta hai aur typically thodi quality loss hoti hai (MQA ke liye ≈1–2% perplexity). Ye ek approximation hai, rewrite nahi.

"FlashAttention aur KV-caching same bottleneck solve karte hain, isliye dono use karna redundant hai."

Ye alag bottlenecks attack karte hain. KV-cache steps ke across redundant

K, V

projection recompute hatata hai; FlashAttention ek step ke andar

O (L^{2})

score matrix ko materialize karne ki memory cost hatata hai. Ye compose karte hain.

"Memory bachane ke liye hum cache ko bina kisi downside ke int8 mein store kar sakte hain."

Flaw "bina kisi downside ke" hai. Cache quantization memory bahut shrink karta hai, lekin low-precision

K, V

har attention score mein rounding error inject karte hain — usually acceptable, lekin free nahi. Ye ek tradeoff hai, freebie nahi.

B

beams ke saath beam search ko greedy decoding jaisa hi cache chahiye."

Galat:

B

beams mein se har ek ek alag hypothesis hai apne past tokens ke saath, isliye aapko

B

alag KV-caches chahiye (ya ek shared-prefix cache per-beam tails ke saath). Memory beam width ke saath scale karta hai.

Why questions

Yahan reasoning hi poora jawab hai.

Naive (no-cache) projection cost

O (L d^{2})

ki jagah

O (L^{2} d^{2})

kyun badhti hai?

Step

t

par naive model sequence ko fresh treat karta hai aur saare

t

tokens ke projections recompute karta hai,

O (t d^{2})

cost karta hai.

\sum_{t = 1}^{L} t \approx L^{2} /2

sum karne se quadratic milta hai. Cache har step ki cost ko constant

O (d^{2})

banata hai, isliye sum linear ho jaata hai.

K

aur

V

cache hote hain, attention output kyun nahi?

K, V

sirf past tokens par depend karte hain aur fixed hain, isliye ye reusable inputs hain. Attention output current

Q_{t}

par depend karta hai, jo har step naya hota hai — isliye output recompute karna hi padta hai aur cache karne ke liye kuch stable nahi hai.

KV-cache size layers ki sankhya ke saath kyun scale karta hai?

Har transformer layer ka apna independent attention hai apne

K, V

projections ke saath. Caching per layer hoti hai, isliye total cache = per-layer cache

\times n_{layers}

. Isliye ek per-token cache ~48 KB (ek layer) se ~4.7 MB (saari layers) tak jump karta hai GPT-3-scale model mein.

MQA inference latency mein raw FLOP count se zyada help kyun karta hai?

Autoregressive generation memory-bandwidth bound hai, compute bound nahi — bottleneck cache ko har step memory se load karna hai. MQA cache ko

h \times

shrink karta hai, isliye per token memory bus ke across bahut kam data jaata hai, jo FLOP savings se bhi bada real-world speedup deta hai.

Hum generation cost amortise karne ke liye batch size indefinitely kyun nahi badha sakte?

Kyunki batch mein har sequence ko apna poora KV-cache chahiye, aur cache memory (batch × length × layers × heads) ke saath badhti hai. Aap jaldi GPU memory ceiling hit karte ho — weights nahi, cache large batches ke liye limiting resource ban jaata hai. deployment considerations dekho.

Caching ke baad bhi causal mask kyun matter karta hai, jab cache sirf past tokens hold karta hai?

Practice mein caches aksar batched aur padded hote hain, aur kuch pipelines temporarily placeholder ya future entries hold karti hain. Mask guarantee hai ki token

t

mathematically sirf

\leq t

ko attend karta hai, generation ko causal rakhta hai chahe buffer mein physically kuch bhi baitha ho.

Edge cases

Boundary conditions jo log bhool jaate hain. Har case ko uski extreme tak push karo.

Bilkul pehle token par (

t = 1

), KV-cache mein kya hai aur kya caching abhi help karta hai?

Token 1 se pehle cache empty hai; aap

K_{1}, V_{1}

compute karte ho aur store karte ho. Pehle se reuse karne ke liye kuch nahi hai, isliye caching step 1 par kuch nahi bachata — fayda step 2 se shuru hota hai.

Length

L = 1

(single-token generation) ki sequence ke liye cache speedup kya hai?

Exactly

1 \times

— koi speedup nahi. Speedup factor

\approx L

hai, isliye

L = 1

ke liye amortise karne ke liye kuch nahi. Caching tabhi payoff karta hai jab sequence badhti hai.

Jab model ko window

W

se purani dependency chahiye toh sliding-window cache ka kya hota hai?

Woh information simply gone hai — key/value evict ho chuki hai. Model ab usse attend nahi kar sakta, isliye us far-back context ki zarurat wala koi bhi jawab degrade ho jaata hai. Ye bounded-memory caching ki fundamental accuracy cost hai.

MQA cache-ratio

\frac{1}{h}

mein, limit

h \to 1

physically kya matlab hai?

h = 1

ke saath sirf ek head hai, isliye standard attention pehle se hi single-KV attention hai aur MQA koi reduction nahi deta (

\frac{1}{1} = 1

). MQA ka fayda sirf tab badhta hai jab head count badhta hai.

GQA mein, group count

g = h

kya correspond karta hai, aur

g = 1

kya correspond karta hai?

g = h

matlab har query head apna KV rakhta hai — ye ordinary multi-head attention hai bina kisi cache saving ke.

g = 1

matlab saare heads ek single KV share karte hain — ye full MQA hai maximum saving ke saath. GQA strictly inke beech rehta hai.

Context length bahut zyada badne par (jaise 1M tokens) KV-caching ka fayda kyun limit hota hai?

Do limits collide karte hain: cache memory linearly badhti hai aur GPU ko exhaust kar sakti hai, aur attention-score step abhi bhi per token

O (t d)

cost karta hai, isliye per-token latency badhti rehti hai. Isliye very-long-context systems caching ke upar sparse attention ya windowing ki taraf jaate hain.

Ek prompt jo generation se pehle ek saath process hota hai ("prefill" phase), kya caching us step ko change karta hai?

Prefill saare prompt tokens ke

K, V

ek parallel pass mein compute karta hai — training ki tarah, prefill ke andar koi reuse exploit karne ke liye nahi hai. Wahan caching ka role simply un

K, V

ko store karna hai taaki subsequent one-token-at-a-time decode phase unhe reuse kar sake.

Recall Quick self-test

$Q$ , $K$ , $V$ mein se kaunsa har generation step par recompute hota hai? ::: Sirf $Q$ — ye encode karta hai naya token kya dhundh raha hai; $K$ aur $V$ cache hote hain kyunki wo fixed past tokens se belong karte hain. Kya cache output numbers change karta hai? ::: Nahi — ye sirf identical values ko recompute karna avoid karta hai, isliye results (bit-)identical hote hain. MQA/GQA kaunsa resource bachata hai, aur kya cost aata hai? ::: KV-cache memory aur memory bandwidth $h$ tak ke factor se bachata hai; thodi model quality ki cost hoti hai.

Related vault topics for definitions used above: 6.1.1-transformer-architecture, 6.1.4-multi-head-attention, 6.1.11-sparse-attention, 6.3.1-model-quantization, 5.4.2-beam-search, 6.2.3-inference-optimization, 7.2.1-llm-deployment, aur parent KV-cache optimization.