Query, key, value matrices

4.1.3 · AI-ML › Transformer Architecture

Q, K, V Matrices Kya Hain?

• Query matrix $Q=X{W}^Q$ jahan $W^Q\in {\mathbb{R}}^{d_{model}\times d_k}$
• Key matrix $K=X{W}^K$ jahan $W^K\in {\mathbb{R}}^{d_{model}\times d_k}$
• Value matrix $V=X{W}^V$ jahan $W^V\in {\mathbb{R}}^{d_{model}\times d_v}$

Ye learned linear transformations hain jo har token ko teen alag subspaces mein project karti hain, jo attention mechanism mein alag-alag purposes ke liye optimize hote hain.

Teen Alag Matrices Kyun?

Alag transformations kyun?

1. Roles ko decouple karna: Ek hi token embedding alag-alag purposes serve karti hai. Jab token A, token B ko attend karta hai, toh humein chahiye:

   • A ki query: "Main kya dhundh raha hoon?"
   • B ki key: "Main matching ke liye kya represent karta hoon?"
   • B ki value: "Main kaunsi information contribute karunga?"

2. Asymmetric relationships: "cat" aur "sat" ke beech ki similarity alag tarike se compute honi chahiye, na ki waisi jaise "cat" information provide karta hai. Keys matching geometry handle karte hain, values information geometry handle karti hain.

3. Optimization flexibility: Alag weight matrices network ko ye seekhne deti hain ki "cat" verbs ke liye query kar sakta hai (Q-K space mein "sat" ke saath high similarity) lekin noun information provide karta hai (V space mein).

First Principles Se Derivation

Starting Point: Hum Kaunsi Problem Solve Kar Rahe Hain?

Hum chahte hain ki har token relevant context ko attend kare. Raw embeddings $x_i\in {\mathbb{R}}^{d_{model}}$ mein saari semantic information mixed hoti hai. Humein ye karna hai:

Step 1: "Relevance" ko similarity score ke roop mein define karo

Token $i$ ke liye token $j$ ki relevance determine karne ke liye hum compute karte hain: ${\text{relevance}}_{i,j}=\text{similarity}(x_i,x_j)$

Lekin raw dot product $x_i\cdot x_j$ bahut rigid hai—ye similarity ke liye wahi dimensions use karta hai jo position, syntax, semantics, etc. encode karti hain.

Ye kyun fail hota hai: Jo dimensions "word meaning" ke liye optimize hain, wo zaruri nahi ki "is pe attend karna chahiye" ke liye bhi optimize hon.

Step 2: Ek specialized "matching space" mein project karo

Iske bajaye, $x_i$ ko ek query vector $q_i$ mein transform karo jo "main kya dhundh raha hoon" represent kare: $q_i=W^Q{x}_i$

jahan $W^Q\in {\mathbb{R}}^{d_k\times d_{model}}$ "search intent" extract karne ke liye seekha jaata hai.

Kyun: Ab $q_i$ ek $d_k$-dimensional space mein rehta hai jo specifically relevance compute karne ke liye optimize hai (typically $d_k<d_{model}$ efficiency aur overfitting rokne ke liye).

Step 3: Comparison targets ko ek "key space" mein project karo

$x_j$ ko ek key vector $k_j$ mein transform karo jo "main kya offer karta hoon" represent kare: $k_j=W^K{x}_j$

Ab relevance aise compute hoti hai: ${\text{relevance}}_{i,j}=q_i\cdot k_j=(W^Q{x}_i{)}^T(W^K{x}_j)$

Alag $W^Q$ aur $W^K$ kyun: Attention directional hai. "The cat sat" → "cat" ka "sat" ko query karna, "sat" ke "cat" ko query karne se alag hai. Asymmetric matrices asymmetric relationships encode karti hain.

Step 4: Information retrieval ko alag karo

Jab hum relevance jaante hain, toh humein information extract karni hoti hai. Lekin token $j$ se jo information chahiye wo shayad alag ho us cheez se jisne use relevant banaya!

$x_j$ ko ek value vector $v_j$ mein transform karo: $v_j=W^V{x}_j$

jahan $W^V\in {\mathbb{R}}^{d_v\times d_{model}}$ "information content" ke liye optimize hai, "matching" ke liye nahi.

Kyun: "The cat sat on the mat" mein, jab "sat" "cat" ko attend karta hai, toh key decide karne mein help karti hai ki "cat" relevant hai (subject-verb relationship), lekin value determine karti hai ki "cat" kya information provide karta hai (semantic features: animal, subject, etc.).

Full Matrix Form

Ek sequence $X=[x_1,x_2,\dots ,x_n{]}^T\in {\mathbb{R}}^{n\times d_{model}}$ ke liye:

Jahan:

• $Q$ ki har row ek token ka query vector hai
• $K$ ki har row ek token ka key vector hai
• $V$ ki har row ek token ka value vector hai
• $W^Q,W^K,W^V$ learned parameters hain jo backpropagation se update hote hain

Attention mein inhe HOW use kiya jaata hai: $\text{Attention}(Q,K,V)=\text{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)V$

Worked Examples

Maan lo $d_k=2$ hai, aur humne ye seekha hai: $W^Q=\left[\begin{array}{cc}0.1& 0.4\\ 0.2& -0.1\\ -0.3& 0.5\\ 0.6& 0.2\end{array}\right],W^K=\left[\begin{array}{cc}-0.2& 0.3\\ 0.5& 0.1\\ 0.4& -0.2\\ -0.1& 0.6\end{array}\right]$

Query compute karo: $q_1=x_1{W}^Q=[0.5,-0.3,0.8,0.2]\left[\begin{array}{cc}0.1& 0.4\\ 0.2& -0.1\\ -0.3& 0.5\\ 0.6& 0.2\end{array}\right]$

Ye step kyun? 4D semantic space se 2D "search" space mein project karna.

Step-by-step multiplication:

• First dimension: $0.5(0.1)+(-0.3)(0.2)+0.8(-0.3)+0.2(0.6)=0.05-0.06-0.24+0.12=-0.13$
• Second dimension: $0.5(0.4)+(-0.3)(-0.1)+0.8(0.5)+0.2(0.2)=0.20+0.03+0.40+0.04=0.67$

$q_1=[-0.13,0.67]$

Key compute karo: $k_1=x_1{W}^K=[0.5,-0.3,0.8,0.2]\left[\begin{array}{ccc}-0.2& 0.3& \\ 0.5& 0.1& \\ 0.4& -0.2-0.1& 0.6\end{array}\right]$

• First dimension: $0.5(-0.2)+(-0.3)(0.5)+0.8(0.4)+0.2(-0.1)=-0.10-0.15+0.32-0.02=0.05$
• Second dimension: $0.5(0.3)+(-0.3)(0.1)+0.8(-0.2)+0.2(0.6)=0.15-0.03-0.16+0.12=0.08$

$k_1=[0.05,0.08]$

Query se alag kyun? Usi token ke alag roles hain: "main kya dhundhta hoon" vs. "main kya offer karta hoon jab koi mujhe dhundhe."

${\text{score}}_{1,1}=q_1\cdot k_1=[-0.13,0.67]\cdot [0.05,0.08]$

Dot product kyun? Attention space mein alignment measure karta hai. High dot product = high relevance.

$=(-0.13)(0.05)+(0.67)(0.08)=-0.0065+0.0536=0.047$

Interpretation: Token 1 ki apne aap ke saath thodi si positive relevance hai. Practice mein, ise $\frac{1}{\sqrt{d_k}}=\frac{1}{\sqrt{2}}\approx 0.707$ se scale kiya jaata aur softmax se pehle doosre tokens ke scores ke saath combine kiya jaata.

Scaling kyun? Jaise $d_k$ badhta hai, dot products magnitude mein badhte hain, softmax ko saturation regions mein push karte hain. Scaling gradient flow maintain karta hai.

$v_1=x_1{W}^V=[0.5,-0.3,0.8,0.2]\left[\begin{array}{ccc}0.3& -0.2& 0.5\\ -0.1& 0.4& 0.2\\ 0.6& 0.1& -0.3\\ 0.2& -0.5& 0.4\end{array}\right]$

• First: $0.5(0.3)+(-0.3(-0.1)+0.8(0.6)+0.2(0.2)=0.15+0.03+0.48+0.04=0.70$
• Second: $0.5(-0.2)+(-0.3)(0.4)+0.8(0.1)+0.2(-0.5)=-0.10-0.12+0.08-0.10=-0.24$
• Third: $0.5(0.5)+(-0.3)(0.2)+0.8(-0.3)+0.2(0.4)=0.25-0.06-0.24+0.08=0.03$

$v_1=[0.70,-0.24,0.03]$

Alag dimension $d_v$ kyun? Values information content encode karti hain, jiske liye matching space se alag dimensionality ki zaroorat ho sakti hai. Simplicity ke liye often $d_v=d_k$ rakha jaata hai, lekin ye zaroori nahi.

HOW use hota hai: Relevance scores ke softmax se attention weights ${\alpha }_{i,j}$ compute karne ke baad, token $i$ ka output hai: ${\text{output}}_i={\sum }_{j=1}^n{\alpha }_{i,j}{v}_j$

Common Mistakes

Kyun sahi lagta hai: Hum architecturally unke dimensions aur roles define karte hain.

Kyun galat hai: $W^Q,W^K,W^V$ learned parameters hain. Ye random start hote hain aur gradient descent se optimize hote hain taaki ye specific task mein attention ke liye relevant features seekh sakein.

Fix: Inhe "learnable lenses" ki tarah socho jo model task-relevant patterns pe focus karne ke liye tune karta hai. Ek translation model mein ye alag patterns seekhte hain, aur ek summarization model mein alag.

Kyun sahi lagta hai: Kabhi kabhi implementation code mein "QKV" ek single matrix ke roop mein dikhta hai.

Kyun galat hai: Q, K, V usi input $X$ ke teen alag linear projections hain. Ye $X$ ko parts mein nahi kaatate; ye poore $X$ ko teen alag tareekon se transform karte hain.

Fix:

• Nahi: $[Q|K|V]=X$ (galat: ye concatenation hai)
• Balki: $Q=X{W}^Q$, $K=X{W}^K$, $V=X{W}^V$ (sahi: teen alag transformations)

Kabhi kabhi implementations $W^{QKV}=[W^Q|W^K|W^V]$ compute karke ek matrix multiply karti hain, phir result split karti hain, lekin conceptually ye alag transformations hain.

Kyun sahi lagta hai: Zyada dimensions = zyada information preserved.

Kyun galat hai:

1. Computational cost: Attention $O(n^2{d}_k)$ hai. $d_k$ reduce karna compute bachata hai.
2. Regularization: Lower-dimensional projections model ko attention ke liye compressed, essential features seekhne par majboor karte hain.
3. Multi-head attention: $h$ heads ke saath, har ek $d_k$ dimension ka, total dimension $h\cdot d_k=d_{model}$ hoti hai, jo heads ke across expressiveness maintain karta hai.

Fix: $d_k$ ko "attention query language" ki dimension socho—ek compressed, task-optimized space. Typical: $d_k=d_{model}/h$ jahan $h$ heads ki sankhya hai.

Q, K, V Multi-Head Attention Ko Kaise Enable Karte Hain

Multi-head attention mein, hum $i=1,\dots ,h$ ke liye $h$ sets of $(W_i^Q,W_i^K,W_i^V)$ banate hain. Har head:

1. $Q_i=X{W}_i^Q$, $K_i=X{W}_i^K$, $V_i=X{W}_i^V$ compute karta hai
2. Attention run karta hai: ${\text{head}}_i=\text{Attention}(Q_i,K_i,V_i)$
3. Heads concatenate aur project hote hain: $\text{MultiHead}(Q,K,V)=\text{Concat}({\text{head}}_1,\dots ,{\text{head}}_h)W^O$

Multiple heads kyun? Alag heads alag attention patterns seekh sakte hain:

• Head 1: syntactic relationships (subject-verb)
• Head 2: semantic similarity (synonyms)
• Head 3: positional patterns (nearby words)

Har head ke Q, K, V matrices alag aspects mein specialize ho jaate hain.

Query (Q): Ye waisa hai jaise tumne haath uthaya ho aur tumhare specific question likha card ho—"Mujhe fractions mein help chahiye!"

Key (K): Ye waisa hai jaise class mein baaki sab cards utha ke dikhaa rahe hon ki wo kisme acche hain—"Main fractions jaanta hoon!" ya "Main spelling mein great hoon!" ya "Main science jaanta hoon!"

Value (V): Ye actual helpful information hai jo log de sakte hain. Sarah ki KEY kehti hai "Main fractions jaanta hoon," lekin uski VALUE actual explanation hai jo wo provide kar sakti hai ki 1/2 aur 1/3 kaise add karte hain.

Teacher (attention mechanism) tumhara QUERY card dekhti hai, usse sabke KEY cards se compare karti hai best matches dhundhne ke liye, phir unse VALUE information collect karti hai aur tumhe combined answer deti hai.

Magical part? Transformer seekhta hai ki ye cards kaise likhne hain! Pehle, wo Q, K, V cards pe random cheezein likhta hai, lekin hazaaron examples dekhne ke baad, wo seekhta hai:

• QUERY cards kaise likhein jo sahi sawaal poochhein
• KEY cards kaise likhein jo relevant help advertise karein
• VALUE cards kaise likhein jo sabse useful information de

Aur sabse cool part ye hai: Usi student (word token) ke paas teeno tarah ke cards hote hain. Jab TUM sawaal poochho, tum apna QUERY card use karte ho. Lekin jab KOI AUR poochhe, wo tumhara KEY card dekhta hai ye jaanne ke liye ki tum help kar sakte ho ya nahi, aur tumhare VALUE card se information leta hai.

Ya: "Query the library, spine pe Keys match karo, andar ki Values padho."

Connections

• Attention Mechanism: Q, K, V matrices attention function ke inputs hain
• Scaled Dot-Product Attention: Attention scores compute karne ke liye $Q{K}^T$ use karta hai
• Multi-Head Attention: Q, K, V projections ke multiple sets banata hai
• Linear Layers: Q, K, V transformations linear layers hain (matrix multiplications)
• Embedding Layer: Input $X$ embedding layer se aata hai
• Positional Encoding: Q, K, V projection se pehle $X$ mein add hota hai
• Self-Attention: Q, K, V sab usi sequence se derive hote hain
• Cross-Attention: Q ek sequence se, K aur V doosre se (encoder-decoder)
• Backpropagation: Jisse $W^Q,W^K,W^V$ seekhe jaate hain
• Gradient Descent: Weight matrices update karne wala optimization algorithm

#flashcards/ai-ml

Attention mein teen matrices kaunsi hain aur conceptually har ek kya represent karti hai? :: Query (Q) = "main kya search kar raha hoon", Key (K) = "main matching ke liye kya offer karta hoon", Value (V) = "main kaunsi information provide karta hoon". Ye input embeddings ke learned linear projections hain.

Input $X\in {\mathbb{R}}^{n\times d_{model}}$ diya gaya, Q, K, V matrices ke formulas likho :: $Q=X{W}^Q$, $K=X{W}^K$, $V=X{W}^V$ jahan $W^Q,W^K\in {\mathbb{R}}^{d_{model}\times d_k}$ aur $W^V\in {\mathbb{R}}^{d_{model}\times d_v}$ learned weight matrices hain.

Concept Map