Rotary positional embeddings (RoPE)

What Problem Does RoPE Solve?

Classic positional encoding (sinusoidal or learned) adds position vectors to token embeddings: $\text{input} = \text{embedding}(x) + \text{pos}(i)$

Problems:

1. Absolute position bias: The model learns "position 0 is special" rather than "these tokens are 3 apart"
2. Length extrapolation fails: Trained on sequences of length 512, it breaks on length 2048
3. Attention doesn't naturally see relative distance: $QK^T$ mixes content + absolute position

RoPE injects position information directly into the attention mechanism by rotating query/key vectors based on their positions, making relative distance fall out naturally from the dot product.

The Mathematics: Deriving RoPE from First Principles

Step 1: What We Want

For tokens at positions $m$ and $n, we want their attention score to depend on **relative position**$m - n$: $\text{score}(m, n) = f(q_m, k_n, m - n)$

Step 2: The Rotation Trick

Consider 2D vectors (we'll generalize). A rotation matrix by angle $\theta$ is: $R(\theta) = \begin{bmatrix} \cos\theta & -\sin\theta \\ \sin\theta & \cos\theta \end{bmatrix}$

Key insight: Rotating a vector $q$ by $\theta_m$ and another vector $k$ by $\theta_n$, then taking their dot product:

Why this step? Orthogonal matrices satisfy $R^T(\alpha) R(\beta) = R(\beta - \alpha)$. The dot product now depends on the angle difference $\theta_n - \theta_m$, not individual angles!

Step 3: Position-Dependent Rotation Angles

Set $\theta_m = m \cdot \theta$ where $\theta$ is a base frequency. Now: $\theta_n - \theta_m = (n - m) \cdot \theta$

The attention score becomes: $q_m^T k_n = (R(m\theta) q)^T (R(n\theta) k) = q^T R((n-m)\theta) k$

The relative position $n - m$ emerges naturally!

Step 4: Generalizing to High Dimensions

Real embedings have dimension $d$ (e.g., 64 for each head). Split $d$ into $d/2$ pairs of dimensions. For pair $i$, use frequency: $\theta_i = 10000^{-2i/d}$

Why this formula? Borrowed from sinusoidal encodings—lower dimensions get higher frequencies (capture fine details), higher dimensions get lower frequencies (capture coarse patterns).

For a vector $x = [x_1, x_2, \ldots, x_d]$ at position $m$, apply rotation to each pair:

Worked Examples

Example 1: 2D Vectors, Positions 0 and 3

Setup: $q = [1, 0]$, $k = [0, 1]$, $\theta = \pi/4$ (45°).

Step 1: Rotate $q$ at position $m=0$: $R_0 q = R(0) q = I q = [1, 0]$

Step 2: Rotate $k$ at position $n=3$: $R_3 k = R(3\pi/4) [0, 1] = \begin{bmatrix} \cos(3\pi/4) & -\sin(3\pi/4) \\ \sin(3\pi/4) & \cos(3\pi/4) \end{bmatrix} \begin{bmatrix} 0 \\ 1 \end{bmatrix}$ $= \begin{bmatrix} -\frac{1}{\sqrt{2}} \ \frac{1}{\sqrt{2}} \end{bmatrix} \cdot 1 + \begin{bmatrix} \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} \end{bmatrix} \cdot 0 = \begin{bmatrix} -1/\sqrt{2} \\ 1/\sqrt{2} \end{bmatrix}$

Step 3: Dot product: $q'^T k' = [1, 0]^T \begin{bmatrix} -1/\sqrt{2} \\ 1/\sqrt{2} \end{bmatrix} = -1/\sqrt{2}$

Why this step? The score encodes that tokens are 3 positions apart via the rotation angle $3\pi/4$.

Example 2: Length Extrapolation

Scenario: Trained on sequences up to length 512. Now process length 2048.

Without RoPE: Absolute position 2000 was never seen during training → model fails.

With RoPE: Take the first dimension pair ($i=1$, $d=64$). Its frequency is $\theta_1 = 10000^{-2/64} = 10000^{-1/32} \approx 0.75$. Position 2000 gives the accumulated angle: $m\theta_1 = 2000 \cdot 0.75 = 1500 \text{ radians} \equiv 1500 \bmod 2\pi \approx 1.86 \text{ radians}$

The model has seen all relative distances from 0 to 512 during training. A token pair at positions (2000, 2005) has relative distance 5, which contributes an angle difference $5\theta_1 = 5 \cdot 0.75 = 3.75$ radians—already learned from any pair 5 apart! The rotation matrices generalize.

Why this works: Rotations wrap around ($2\pi$ periodicity), so unseen absolute positions still produce familiar relative angle differences.

Diagram

Common Mistakes

Implementation Details

Complex number trick: Represent each2D rotation as multiplication by $e^{im\theta} = \cos(m\theta) + i\sin(m\theta)$. The vector $[x_1, x_2]$ becomes complex number $x_1 + ix_2$, and rotation is just: $(x_1 + ix_2) \cdot e^{im\theta}$$

Convert back to real/imaginary parts for the rotated coordinates. This is how libraries like transformers implement it.

Connections to Other Concepts

• Sinusoidal Positional Encoding: RoPE uses the same frequency schedule but applies it as rotations instead of additions
• Multi-Head Attention: Each head can apply RoPE independently, allowing different heads to specialize in different position scales
• Relative Position Bias (T5): Both encode relative position, but RoPE does it implicitly through rotation geometry rather than explicit bias terms
• Attention Mechanism: RoPE modifies the $QK^T$ computation to inject position information
• ALiBi (Attention with Linear Biases): Alternative approach that adds position bias directly to attention scores rather than rotating embedings
• Length Extrapolation in LLMs: RoPE is one of the key techniques enabling models to handle sequences longer than training length

Active Recall Practice

Flashcards

#flashcards/ai-ml

What is the key advantage of RoPE over additive positional encodings? :: RoPE encodes relative positions naturally through rotation angle differences, enabling better length extrapolation and making attention scores depend on token distance rather than absolute positions.

Concept Map

Hinglish (regional understanding)