Rotary positional embeddings (RoPE): Foundations

This page assumes you know nothing. We build every symbol the parent note (see the Hinglish version too) leans on, one brick at a time, each brick resting on the one before.

0. What a vector is (the very first brick)

The two numbers are called components. $x_{1}$ is how far right, $x_{2}$ is how far up. That is the whole picture — and Figure s01 draws exactly this arrow so you can attach the words to the geometry.

Figure — Rotary positional embeddings (RoPE)

In Figure s01 the blue arrow ends at $(3, 2)$ ; the yellow dashed line is its $x_{1} = 3$ (rightness) and the red dashed line is its $x_{2} = 2$ (upness). Whenever this note says "vector," picture that arrow.

Why does RoPE need this? Because a token's meaning is stored as a longer such list (dimension $d$ , e.g. 64 numbers). Everything RoPE does is geometry on these arrows.

1. Position indices $m$ and $n$

A token is one chunk of the input text (a word or word-piece) that the model processes as a unit.

Picture a row of boxes numbered $0, 1, 2, 3, \dots$ — each box holds one token.

1b. What queries and keys are

2. Angles, and why we measure them in radians

An angle measures turning. We could use degrees (a full turn = $360°$ ), but maths and code prefer radians.

Key fact we will reuse: angles wrap. Turning by $1500$ radians lands you in the same spot as turning by $1500 mod 2 π \approx 4.60$ radians, because every extra $2 π$ is one full circle back home.

3. Sine and cosine — the coordinates of a spun point

Put a point on a circle of radius 1 (the unit circle). Sweep an angle $θ$ counter-clockwise from the positive $x$ -axis. Where did the point land?

Figure s02 draws this: the green arc is the swept angle $θ = 3 π /4$ , the yellow segment is $cos θ$ (here pointing left, so negative) and the red segment is $sin θ$ (pointing up, positive).

Why does RoPE need $cos$ and $sin$ ? Because "spinning a vector" means computing where its tip goes on a circle — and $cos$ / $sin$ are exactly the machine that reports those new coordinates. Every quadrant is covered by the same two functions:

Quadrant	angle range	$cos θ$	$sin θ$
I	$0$ to $\frac{π}{2}$	$+$	$+$
II	$\frac{π}{2}$ to $π$	$-$	$+$
III	$π$ to $\frac{3 π}{2}$	$-$	$-$
IV	$\frac{3 π}{2}$ to $2 π$	$+$	$-$

Notice $θ = 3 π /4$ (which we will use in the worked check at the end, and which Figure s02 draws) sits in Quadrant II: $cos$ negative, $sin$ positive. We will confirm the exact numbers below.

4. Rotating a vector — the matrix that spins arrows

We have an arrow $[x_{1}, x_{2}]$ and want a new arrow that is the old one turned by angle $θ$ , same length, new direction.

Figure s03 shows this in action: the blue arrow is the original $x$ , the yellow arrow is $R (θ) x$ , and the green arc is the turn $θ$ between them — same length, new direction.

Sanity check the four cases:

$θ = 0$ : $R = [1001]$ , the identity — no turn (this is why a token at position 0 comes out unchanged).
$θ = π /2$ : $[1, 0] \to [0, 1]$ — right-arrow becomes up-arrow. Correct.
$θ = π$ : every arrow flips to its opposite.
Negative $θ$ : turns clockwise.

5. The dot product — how we measure "agreement"

Attention compares a query $q$ with a key $k$ (both defined in §1b). The comparison tool is the dot product.

Why RoPE needs it: the raw attention score between a query and key is a dot product. If we can make that number depend only on $m - n$ , we have won.

6. The magic property: spins subtract

Here is the fact the whole method rests on.

Now the identity we need: $R (α)^{T} R (β) = R (- α) R (β) = R (β - α)$

WHAT we did: took $(R (α) q)^{T} (R (β) k) = q^{T} R (α)^{T} R (β) k = q^{T} R (β - α) k$ . WHY: because $R (α)^{T} = R (- α)$ undoes part of $R (β)$ 's turn, leaving the net turn $β - α$ . WHAT IT LOOKS LIKE: two arrows each spun on their own dials, but the angle between them only knows the difference of the dial readings.

Set $α = m θ$ and $β = n θ$ and you get $β - α = (n - m) θ$ — the relative position drops out. That single line is RoPE.

7. The frequency schedule $θ_{i} = 1000 0^{- 2 i / d}$

Real vectors have $d$ numbers, not 2. RoPE chops them into $d /2$ pairs and spins each pair by its own angle-per-step $θ_{i}$ .

Figure s04 plots $θ_{i}$ against pair index $i$ on a log axis: early pairs (small $i$ ) sit high — fast dials; later pairs (large $i$ ) sink toward zero — slow dials.

Intuition Why the exponent

$- 2 i / d$ , and why 10000? As $i$ runs from $1$ to $d /2$ , the quantity $2 i / d$ runs from $2/ d$ up to $1$ . So the exponent $- 2 i / d$ runs from $- 2/ d$ (just below $0$ ) down to $- 1$ . That makes $θ_{i}$ range from $1000 0^{- 2/ d}$ — close to $1$ but not exactly $1$ (for $d = 64$ that is $\approx 0.75$ , the fastest dial) — down to $1000 0^{- 1} = 0.0001$ (the slowest dial). The speeds form a geometric ladder evenly spaced across many orders of magnitude. The base 10000 is inherited from the original sinusoidal positional encoding: it is large enough that the slowest dial's period ( $\approx 2 π \cdot 10000$ positions) comfortably exceeds any realistic sequence length, so even the longest-range dial has not yet wrapped around. It is a convention, not a magic constant — other bases work, and some long-context models deliberately raise it.

Prerequisite map

Each foundation on the left feeds the box to its right; all roads end at RoPE, which then powers Multi-Head Attention and rivals ALiBi (Attention with Linear Biases) and Relative Position Bias (T5).

Tiny worked check (spin one key, then dot)

Let us run the machinery on the smallest possible case so every symbol earns its keep. Take a query $q = [1, 0]$ at position $m = 0$ and a key $k = [0, 1]$ at position $n = 3$ , with per-step angle $θ = π /4$ .

Step A — spin the query. At position $m = 0$ the angle is $m θ = 0$ , so $R (0) = I$ leaves it alone: $q^{'} = [1, 0]$ . (Recall from §1b that the prime just means "after rotation.")

Step B — spin the key. At position $n = 3$ the angle is $n θ = 3 π /4$ : $k^{'} = R (3 π /4) [01] = [cos \frac{3 π}{4} \cdot 0 - sin \frac{3 π}{4} \cdot 1 sin \frac{3 π}{4} \cdot 0 + cos \frac{3 π}{4} \cdot 1] = [- \frac{1}{2} - \frac{1}{2}]$ Here $cos \frac{3 π}{4} = - \frac{1}{2}$ and $sin \frac{3 π}{4} = + \frac{1}{2}$ (Quadrant II, as the table in §3 predicts), so the first entry is $- sin \frac{3 π}{4} = - \frac{1}{2}$ and the second is $cos \frac{3 π}{4} = - \frac{1}{2}$ .

Step C — dot them. The attention score is $q^{' T} k^{'} = [1, 0] [- \frac{1}{2} - \frac{1}{2}] = 1 \cdot (- \frac{1}{2}) + 0 \cdot (- \frac{1}{2}) = - \frac{1}{2} \approx - 0.707.$ By §6, this equals $q^{T} R ((n - m) θ) k = q^{T} R (3 π /4) k$ — the score depends only on the gap $n - m = 3$ , exactly as promised.

The wrap-around fact from §2 in numbers: $1500 mod 2 π \approx 4.60$ radians, confirming that a huge position angle folds back into a familiar small one inside $[0, 2 π)$ .

Equipment checklist

Test yourself — cover the right side.

A vector in 2D is... ::: an ordered pair $[x_{1}, x_{2}]$ , drawn as an arrow from the origin.
A token is... ::: one chunk of input text (word or word-piece) the model processes as a unit.
A query and a key are... ::: two vectors made from a token; $q$ ("what I seek") is compared against $k$ ("what I offer") to score attention.
A prime like $q^{'}$ means... ::: the vector after we rotated (spun) it, $q^{'} = R q$ .
$m$ and $n$ mean... ::: the absolute positions (box numbers) of two tokens; $m - n$ is their relative distance.
One full turn in radians is... ::: $2 π \approx 6.28$ .
$cos θ$ and $sin θ$ are... ::: the horizontal and vertical coordinates of a point spun by $θ$ on the unit circle.
The rotation matrix is... ::: $[cos θ sin θ - sin θ cos θ]$ .
$R (0)$ equals... ::: the identity matrix — no rotation.
The dot product $q^{T} k$ equals... ::: $q_{1} k_{1} + q_{2} k_{2}$ , a single number measuring alignment.
"Orthogonal matrix" means... ::: $M^{T} M = I$ ; it preserves lengths and angles, and for rotations $R (θ)^{T} = R (- θ)$ .
The magic identity is... ::: $R (α)^{T} R (β) = R (β - α)$ , so absolute spins cancel and only the difference survives.
RoPE pairs coordinates as... ::: adjacent pairs $(x_{2 i - 1}, x_{2 i})$ : $(x_{1}, x_{2}), (x_{3}, x_{4}), \dots$
The frequency schedule is... ::: $θ_{i} = 1000 0^{- 2 i / d}$ ; small $i$ = fast dial, large $i$ = slow dial; $d$ must be even.
Why RoPE only rotates Q and K... ::: because $Q K^{T}$ decides where to attend (position matters); values carry content and stay untouched.