Rotary positional embeddings (RoPE)

4.1.7 · AI-ML › Transformer Architecture

What Problem Does RoPE Solve?

Classic positional encoding (sinusoidal ya learned) position vectors ko token embeddings mein add karta hai: $\text{input}=\text{embedding}(x)+\text{pos}(i)$

Problems:

1. Absolute position bias: Model "position 0 special hai" seekhta hai, yeh nahin ki "yeh tokens 3 apart hain"
2. Length extrapolation fail ho jaati hai: Length 512 ke sequences par train karo, length 2048 par toot jaata hai
3. Attention naturally relative distance nahin dekhta: $Q{K}^T$ content + absolute position dono ko mix karta hai

RoPE position information ko seedha attention mechanism mein inject karta hai query/key vectors ko unki positions ke basis par rotate karke, jisse relative distance dot product se naturally nikal aati hai.

The Mathematics: Deriving RoPE from First Principles

Step 1: What We Want

Positions $m$ aur $n$ par tokens ke liye, hum chahte hain ki unka attention score relative position $m-n$ par depend kare: $\text{score}(m,n)=f(q_m,k_n,m-n)$

Step 2: The Rotation Trick

2D vectors consider karo (baad mein generalize karenge). Angle $\theta$ ke liye rotation matrix hai: $R(\theta )=\left[\begin{array}{cc}\cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{array}\right]$

Key insight: Vector $q$ ko ${\theta }_m$ se aur vector $k$ ko ${\theta }_n$ se rotate karo, phir unka dot product lo:

Yeh step kyun? Orthogonal matrices satisfy karte hain $R^T(\alpha )R(\beta )=R(\beta -\alpha )$. Ab dot product angle difference ${\theta }_n-{\theta }_m$ par depend karta hai, individual angles par nahin!

Step 3: Position-Dependent Rotation Angles

${\theta }_m=m\cdot \theta$ set karo jahan $\theta$ ek base frequency hai. Ab: ${\theta }_n-{\theta }_m=(n-m)\cdot \theta$

Attention score ban jaata hai: $q_m^T{k}_n=(R(m\theta )q{)}^T(R(n\theta )k)=q^{T}R((n-m)\theta )k$

Relative position $n-m$ naturally saamne aa jaata hai!

Step 4: Generalizing to High Dimensions

Real embeddings ki dimension $d$ hoti hai (jaise har head ke liye 64). $d$ ko $d/2$ dimension pairs mein split karo. Pair $i$ ke liye, frequency use karo: ${\theta }_i=10000^{-2i/d}$

Yeh formula kyun? Sinusoidal encodings se liya gaya hai—lower dimensions ko higher frequencies milti hain (fine details capture karte hain), higher dimensions ko lower frequencies (coarse patterns capture karte hain).

Position $m$ par vector $x=[x_1,x_2,\dots ,x_d]$ ke liye, har pair par rotation apply karo:

Worked Examples

Example 1: 2D Vectors, Positions 0 aur 3

Setup: $q=[1,0]$, $k=[0,1]$, $\theta =\pi /4$ (45°).

Step 1: $q$ ko position $m=0$ par rotate karo: $R_{0}q=R(0)q=Iq=[1,0]$

Step 2: $k$ ko position $n=3$ par rotate karo: $R_{3}k=R(3\pi /4)[0,1]=\left[\begin{array}{cc}\cos (3\pi /4)& -\sin (3\pi /4)\\ \sin (3\pi /4)& \cos (3\pi /4)\end{array}\right]\left[\begin{array}{c}0\\ 1\end{array}\right]$ $=\left[\begin{array}{c}-\frac{1}{\sqrt{2}}\frac{1}{\sqrt{2}}\end{array}\right]\cdot 1+\left[\begin{array}{c}\frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}}\end{array}\right]\cdot 0=\left[\begin{array}{c}-1/\sqrt{2}\\ 1/\sqrt{2}\end{array}\right]$

Step 3: Dot product: $q^{\prime T}{k}^{\prime }=[1,0{]}^T\left[\begin{array}{c}-1/\sqrt{2}\\ 1/\sqrt{2}\end{array}\right]=-1/\sqrt{2}$

Yeh step kyun? Score encode karta hai ki tokens rotation angle $3\pi /4$ ke through 3 positions apart hain.

Example 2: Length Extrapolation

Scenario: Length 512 tak ke sequences par train kiya. Ab length 2048 process karo.

RoPE ke bina: Absolute position 2000 training mein kabhi nahin dekha gaya → model fail ho jaata hai.

RoPE ke saath: Pehla dimension pair lo ($i=1$, $d=64$). Iski frequency hai ${\theta }_1=10000^{-2/64}=10000^{-1/32}\approx 0.75$. Position 2000 yeh accumulated angle deta hai: $m{\theta }_1=2000\cdot 0.75=1500\text{ radians}\equiv 1500\mathrm{mod}2\pi \approx 1.86\text{ radians}$

Model ne training mein saare relative distances 0 se 512 tak dekhe hain. (2000, 2005) positions par ek token pair ki relative distance 5 hai, jo angle difference $5{\theta }_1=5\cdot 0.75=3.75$ radians contribute karta hai—yeh pehle se kisi bhi 5-apart pair se seekha ja chuka hai! Rotation matrices generalize karte hain.

Yeh kyun kaam karta hai: Rotations wrap around ho jaati hain ($2\pi$ periodicity), isliye unseen absolute positions phir bhi familiar relative angle differences produce karte hain.

Diagram

Common Mistakes

Implementation Details

Complex number trick: Har 2D rotation ko $e^{im\theta }=\cos (m\theta )+i\sin (m\theta )$ se multiplication ke roop mein represent karo. Vector $[x_1,x_2]$ complex number $x_1+i{x}_2$ ban jaata hai, aur rotation sirf hai: $(x_1+i{x}_2)\cdot e^{im\theta }$

Rotated coordinates ke liye real/imaginary parts mein wapas convert karo. transformers jaisi libraries mein isi tarah implement kiya jaata hai.

Connections to Other Concepts

• Sinusoidal Positional Encoding: RoPE wohi frequency schedule use karta hai lekin ise additions ke bajaye rotations ke roop mein apply karta hai
• Multi-Head Attention: Har head RoPE independently apply kar sakta hai, allowing different heads ko different position scales mein specialize karne ke liye
• Relative Position Bias (T5): Dono relative position encode karte hain, lekin RoPE yeh rotation geometry ke through implicitly karta hai, explicit bias terms ke bajaye
• Attention Mechanism: RoPE $Q{K}^T$ computation ko position information inject karne ke liye modify karta hai
• ALiBi (Attention with Linear Biases): Alternative approach jo attention scores mein seedha position bias add karta hai, embeddings ko rotate karne ke bajaye
• Length Extrapolation in LLMs: RoPE un key techniques mein se ek hai jo models ko training length se zyada lambe sequences handle karne mein enable karti hai

Active Recall Practice

Flashcards

#flashcards/ai-ml

What is the key advantage of RoPE over additive positional encodings? :: RoPE relative positions ko naturally rotation angle differences ke through encode karta hai, better length extrapolation enable karta hai aur attention scores ko absolute positions ki jagah token distance par depend karta hai.

Concept Map