Positional encodings (sinusoidal)

4.1.6 · AI-ML › Transformer Architecture

Transformers saare tokens ko parallel mein process karte hain, jisse woh sequential order information kho dete hain jo RNNs naturally preserve karti hai. Positional encoding position information ko token embeddings mein inject karti hai taaki model ko pata chale "kaun sa word pehle aaya."

$P{E}_{(pos,2i)}=\sin \left(\frac{pos}{10000^{2i/d_{model}}}\right)$

$P{E}_{(pos,2i+1)}=\cos \left(\frac{pos}{10000^{2i/d_{model}}}\right)$

Even dimensions sine use karti hain, odd dimensions cosine. Inhe token embeddings mein add kiya jaata hai (concatenate nahi).

Derivation: Yeh Formula Kyun?

Step 1: Kuch aisa chahiye jo position ke saath change ho Hum chahte hain ki $PE(pos)$ har position ke liye unique ho. Ek simple counter $[1,2,3,...]$ kaam karta lekin yeh unboundedly scale karta—position 1000 embedding values par dominate kar leta.

Step 2: Bounded functions—sine aur cosine Trig functions $[-1,1]$ mein oscillate karti hain, values bounded rehti hain. Ek single frequency $\omega$ ke liye:

$PE(pos)=\sin (\omega \cdot pos)$

Yeh step kyun? Sine hume periodicity deta hai. Alag-alag positions ko alag-alag values milti hain, lekin function repeat hota hai. Ek frequency kaafi nahi hai kyunki yeh repeat hoti hai (period $2\pi /\omega$), jisse door positions similar lagti hain.

Step 3: Alag-alag dimensions ke liye multiple frequencies Har embedding dimension ke liye alag frequency use karo. Dimension $i$ ko frequency milti hai:

${\omega }_i=\frac{1}{10000^{2i/d_{model}}}$

Frequencies decreasing kyun?

• Low dimensions ($i$ chhota): high frequency $\to$ position ke saath rapidly change hota hai $\to$ fine-grained local order capture karta hai
• High dimensions ($i$ bada): low frequency $\to$ slowly change hota hai $\to$ coarse global position capture karta hai

Factor $10000$ wavelength range set karta hai. Sabse lamba wavelength $2\pi \cdot 10000\approx 62832$ positions hai.

Step 4: Alternating sine aur cosine Even dimensions sine use karti hain, odd dimensions same frequency ka cosine use karti hain:

$P{E}_{(pos,2i)}=\sin \left(\frac{pos}{10000^{2i/d_{model}}}\right)$ $P{E}_{(pos,2i+1)}=\cos \left(\frac{pos}{10000^{2i/d_{model}}}\right)$

Dono kyun? Pair $(\sin (\omega pos),\cos (\omega pos))$ frequency $\omega$ ke circle par position ki 2D representation banata hai. Isse model trig identities use karke relative positions compute karne ke liye linear transformations seekh sakta hai:

$\sin (\omega (pos+k))=\sin (\omega pos)\cos (\omega k)+\cos (\omega pos)\sin (\omega k)$

Ek linear layer weights $\cos (\omega k)$ aur $\sin (\omega k)$ seekh sakti hai taaki kisi bhi position ko $k$ steps shift kiya ja sake.

Relative Position: Kisi bhi fixed offset $k$ ke liye, $PE(pos+k)$, $PE(pos)$ ka ek linear function hai:

$PE(pos+k)={\mathbf{T}}_k\cdot PE(pos)$

jahan ${\mathbf{T}}_k$ ek linear transformation hai. Isse relative distances learnable ho jaati hain.

Bounded Values: Saare components $[-1,1]$ mein rehte hain.

Extrapolation: Training sequences se lambi positions ke liye bhi kaam karta hai (unlike learned positional embeddings jo max length par cap hoti hain).

Worked Example 1: 4-Dimensional Model ke liye PE Compute Karna

$d_{model}=4$ lo, $PE(pos=3)$ compute karo.

Dimensions: $i\in \{0,1\}$ (kyunki $2i$ aur $2i+1$ indices 0,1,2,3 cover karte hain).

Dimension 0 (i=0, even → sine): $P{E}_{(3,0)}=\sin \left(\frac{3}{10000^{0/4}}\right)=\sin (3)=0.1411$

Dimension 1 (i=0, odd → cosine): $P{E}_{(3,1)}=\cos \left(\frac{3}{10000^{0/4}}\right)=\cos (3)=-0.9900$

Dimension 2 (i=1, even → sine): $\omega =\frac{1}{10000^{2/4}}=\frac{1}{100}=0.01$ $P{E}_{(3,2)}=\sin (3\cdot 0.01)=\sin (0.03)=0.0300$

Dimension 3 (i=1, odd → cosine): $P{E}_{(3,3)}=\cos (3\cdot 0.01)=\cos (0.03)=0.9996$

Yeh steps kyun? Lower dimensions (0,1) rapidly oscillate karti hain (high $\omega =1$), exact position capture karti hain. Higher dimensions (2,3) slowly oscillate karti hain (low $\omega =0.01$), sequence mein coarse position capture karti hain.

Final: $PE(3)=[0.1411,-0.9900,0.0300,0.9996]$

Worked Example 2: Relative Position Linearity Demonstrate Karna

Dikhao ki $PE(pos+1)$, $PE(pos)$ se linearly related hai dimension pair $(2i,2i+1)$ ke liye.

Dimension $i$ ke liye, ${\omega }_i=1/10000^{2i/d_{model}}$ lo. Hamare paas hai:

$P{E}_{(pos,2i)}=\sin ({\omega }_i\cdot pos),P{E}_{(pos,2i+1)}=\cos ({\omega }_i\cdot pos)$

Position $pos+1$ ke liye:

$P{E}_{(pos+1,2i)}=\sin ({\omega }_i(pos+1))=\sin ({\omega }_{i}pos+{\omega }_i)$

Angle addition use karke: $=\sin ({\omega }_{i}pos)\cos ({\omega }_i)+\cos ({\omega }_{i}pos)\sin ({\omega }_i)$ $=P{E}_{(pos,2i)}\cdot \cos ({\omega }_i)+P{E}_{(pos,2i+1)}\cdot \sin ({\omega }_i)$

Yeh step kyun? Trig identity nayi position ko purani position ki encoding ke weighted sum mein convert karti hai. Weights $\cos ({\omega }_i)$ aur $\sin ({\omega }_i)$ sirf offset (1 step) par depend karte hain, absolute position par nahi.

Similarly cosine dimension ke liye: $P{E}_{(pos+1,2i+1)}=\cos ({\omega }_{i}pos)\cos ({\omega }_i)-\sin ({\omega }_{i}pos)\sin ({\omega }_i)$ $=P{E}_{(pos,2i+1)}\cdot \cos ({\omega }_i)-P{E}_{(pos,2i)}\cdot \sin ({\omega }_i)$

Matrix form mein: $\left[\begin{array}{c}P{E}_{(pos+1,2i)}\\ P{E}_{(pos+1,2i+1)}\end{array}\right]=\left[\begin{array}{cc}\cos ({\omega }_i)& \sin ({\omega }_i)\\ -\sin ({\omega }_i)& \cos ({\omega }_i)\end{array}\right]\left[\begin{array}{c}P{E}_{(pos,2i)}\\ P{E}_{(pos,2i+1)}\end{array}\right]$

Yeh ek rotation matrix hai! +1 offset 2D representation ko angle ${\omega }_i$ se rotate karta hai. Model yeh transformation seekh sakta hai taaki relative positions par attend kiya ja sake.

Worked Example 3: Base 10000 Kyun?

Dimension $i$ ka wavelength hai:

${\lambda }_i=2\pi \cdot 10000^{2i/d_{model}}$

$d_{model}=512$ ke liye:

• Dimension 0 ($i=0$): ${\lambda }_0=2\pi \cdot 10000^0=2\pi \approx 6.28$ (high frequency)
• Dimension 255 ($i=255$): ${\lambda }_{255}=2\pi \cdot 10000^{510/512}\approx 2\pi \cdot 9930\approx 62393$ (low frequency)

(Note: $10000^{510/512}$ thoda $10000$ se kam hai, isliye maximum wavelength $2\pi \cdot 10000\approx 62832$ se thodi kam hai.)

Yeh range kyun?

• Short wavelengths ($\approx 6.28$) adjacent positions distinguish karte hain (har position alag dikhti hai)
• Long wavelengths ($\approx 62393$) lambe documents mein global position encode karte hain (sequences ~10k tokens tak)

Geometric progression $10000^{2i/d_{model}}$ scales mein smooth coverage ensure karti hai—har dimension position ka ek specific "zoom level" handle karta hai.

Yeh step kyun? Bahut chhota base (jaise 100) → sabse lamba wavelength sirf ~628 positions → lambi sequences par fail. Bahut bada → spectrum bahut zyada stretch ho jaata hai, resolution waste hoti hai. Value $10000$ fine local resolution aur long-range coverage ke beech balance banata hai.

Step 1: Tokens ko vectors mein embed karo (embedding table se):

• "The" → $[0.2,-0.5,0.8,0.1]$
• "cat" → $[-0.3,0.4,0.2,-0.6]$
• "sat" → $[0.7,0.1,-0.4,0.3]$

Step 2: Positions 0,1,2 ke liye positional encodings compute karo ($d_{model}=4$ use karke):

$pos=0$ ke liye:

• $P{E}_{(0,0)}=\sin (0)=0$
• $P{E}_{(0,1)}=\cos (0)=1$
• $P{E}_{(0,2)}=\sin (0)=0$
• $P{E}_{(0,3)}=\cos (0)=1$
• $PE(0)=[0,1,0,1]$

$pos=1$ ke liye:

• $P{E}_{(1,0)}=\sin (1)=0.8415$
• $P{E}_{(1,1)}=\cos (1)=0.5403$
• $P{E}_{(1,2)}=\sin (0.01)=0.0100$
• $P{E}_{(1,3)}=\cos (0.01)=0.99995$
• $PE(1)=[0.8415,0.5403,0.0100,0.99995]$

$pos=2$ ke liye:

• $PE(2)=[0.9093,-0.4161,0.0200,0.9998]$

Step 3: Embeddings mein positional encodings add karo:

• "The" + $PE(0)$ = $[0.2,0.5,0.8,1.1]$
• "cat" + $PE(1)$ = $[0.5415,0.9403,0.2100,0.39995]$
• "sat" + $PE(2)$ = $[1.6093,-0.3161,-0.3800,1.2998]$

Yeh position-aware embeddings pehle transformer layer mein jaati hain.

Yeh step kyun? Addition (concatenation nahi) dimensionality preserve karta hai aur model ko semantic content (embeddings) aur positional information (encodings) blend karne deta hai.

Kyun sahi lagta hai: Kyunki hum PE per position compute karte hain, aisa lagta hai jaise yeh token khud ko encode kar raha hai.

Fix: Positional encoding SIRF yeh encode karta hai ki token kahan hai, kya hai nahi. Token embedding meaning encode karta hai ("cat" vs "dog"). Inhe saath add kiya jaata hai:

$\text{Input}=\text{Embedding}(\text{token})+PE(\text{position})$

Embedding ke bina, model ko bilkul nahi pata ki words ka matlab kya hai. PE ke bina, model ko bilkul nahi pata ki unka order kya hai.

Kyun sahi lagta hai: Concatenation information ko distinct rakhta hai: $[embedding;PE]$ mein dono components intact hain.

Fix: Concatenation dimensionality double kar deta hai ($d_{model}$ → $2\cdot d_{model}$), compute cost badh jaata hai. Addition kaam karta hai kyunki:

1. Embeddings aur PE same $d_{model}$ space mein rehte hain
2. Self-attention learned projections $W_Q,W_K,W_V$ ke zariye inhe disentangle karna seekhta hai
3. PE values chhoti hoti hain (bounded in [-1,1]) embeddings ke comparison mein (larger variance se initialize ki gayi), isliye yeh "nudge" karti hain overwhelm nahi karti

Empirically, addition concatenation jitna hi kaam karta hai, aadhe parameters ke saath.

Kyun sahi lagta hai: Index $i$, 0 se $d_{model}-1$ tak jaata hai, same jaise positions 0 se $d_{model}-1$ tak.

Fix: Har dimension $i$ ek frequency component encode karta hai jo SABHI positions par present hai. Dimension 0 har position par quickly oscillate karta hai (high frequency). Dimension 511 har position par slowly oscillate karta hai (low frequency). Ise Fourier transform ki tarah socho: low frequencies big picture capture karti hain, high frequencies details capture karti hain. Har position saari dimensions use karti hai.

Ek Transformer ek super-smart reader ki tarah hai jo saare panels ek saath dekhta hai. Lekin agar hum isse panels bina numbers ke dein, toh yeh story sahi se nahi bata sakta. Isliye hum har panel par ek secret code likhte hain jo kehta hai "yeh panel 3 hai" bina actually "3" likhe.

Yeh code waves ka pattern hai—jaise wiggly lines se bani fingerprint. Panel 1 ko ek fingerprint milti hai, panel 2 ko doosri, aur aise hi aage. Fingerprint ke kuch hisse fast wiggle karte hain (paas wale panels alag karne ke liye), aur kuch slow wiggle karte hain (yeh batane ke liye ki tum comic ki shuruaat mein ho ya ant mein). Hum yeh fingerprint har panel mein add karte hain, taaki Transformer order jaane jab tak saari pictures ek saath dekhe.

• Fixed (learned nahi)
• Addition to embeddings
• Sine on even, cosine on odd
• Ten-thousand base (10000)

Visual: Ek spiral notebook imagine karo jahan har coil (dimension) ki tightness alag hai—tight coils fine position ke liye, loose coils coarse position ke liye.

Connections

• Token Embeddings: PE inhe add hota hai taaki position-aware inputs bane
• Self-Attention Mechanism: PE attention ko "pichle word par attend karo" aur "agle word par attend karo" mein distinguish karne deta hai
• Multi-Head Attention: Alag-alag heads alag-alag relative position offsets par focus karna seekh sakte hain
• Learned Positional Embeddings: Sinusoidal ka alternative; learned lookup table (training sequence length tak limited)
• Relative Positional Encodings: Modern alternative (T5, Transformer-XL) jo directly attention mein relative distances encode karta hai
• Fourier Transform: Sinusoidal PE position ko frequency components mein decompose karta hai, frequency analysis se analogous
• Rotary Position Embeddings (RoPE): Modern variant jo embeddings mein add karne ki jagah attention space mein rotation apply karta hai

#flashcards/ai-ml

Concept Map