Rotary positional embeddings (RoPE): Visual walkthrough

This is a companion to Rotary positional embeddings (RoPE). We assume you know what a query and key are from Attention Mechanism and Multi-Head Attention — nothing else.

Step 1 — An arrow on a grid: what a 2D vector even is

WHAT. Before any rotation, we need to agree what we are rotating. A vector here is just an arrow drawn from the origin (the point $(0, 0)$ ) to some point on a flat grid. Two numbers pin it down: how far right, how far up.

WHY. RoPE works on pairs of embedding dimensions. So the whole story lives in a flat 2D world first — one pair of numbers, one arrow. Master this picture and the full 64-dimensional case is just this same picture stacked many times.

PICTURE. Look at the two chalk arrows below.

Figure — Rotary positional embeddings (RoPE)

Step 2 — The dot product measures alignment, not position

WHAT. Attention scores one token against another with a dot product. Here $q$ is the query arrow (the token doing the looking) and $k$ is the key arrow (the token being looked at) — both are ordinary 2D vectors exactly like Step 1, with $k = [k_{x} k_{y}]$ ( $k_{x}$ = its rightward amount, $k_{y}$ = its upward amount). For these two arrows the dot product is: $q \cdot k = q_{x} k_{x} + q_{y} k_{y} .$

WHY THIS TOOL and not another? We need one number that says "how much do these two arrows point the same way?" Multiplying matching parts and summing is the simplest such measure — and it has a beautiful geometric identity: $q \cdot k = ∣ q ∣ ∣ k ∣ cos φ,$ where $φ$ (Greek letter phi, just a name for the angle between the two arrows). The $cos φ$ (cosine — the horizontal reach of a unit arrow tilted by $φ$ ) is the star: it is $1$ when the arrows agree, $0$ when perpendicular, $- 1$ when opposite.

PICTURE. The angle $φ$ between the query arrow $q$ and key arrow $k$ is the whole game.

Hold that last bullet. If we can control $φ$ , we control the score.

Step 3 — Rotation: spinning an arrow without stretching it

WHAT. A rotation by angle $α$ takes an arrow and turns it counter-clockwise by $α$ , keeping its length fixed. In numbers, the new arrow is $R (α) q$ where $R (α) = [cos α sin α - sin α cos α] .$

WHY THIS TOOL? We want to stamp a position onto an arrow without changing what it "is" (its length, its content). Rotation is the unique move that changes direction only. Adding a vector (the old way) changes both direction and length — smearing content and position together. Rotation keeps them clean.

PICTURE. Watch one arrow before and after a $60°$ turn — same length, new heading.

Step 4 — Two rotations meet: the angles subtract

WHAT. This is the heart. Rotate $q$ by $α$ , rotate $k$ by $β$ , then dot them: $(R (α) q) \cdot (R (β) k) = q \cdot R (β - α) k .$ The two absolute angles $α, β$ collapse into a single difference $β - α$ .

WHY. From Step 2, the dot product only cares about the angle between the two arrows. Turning $q$ by $α$ and $k$ by $β$ changes the gap between them by exactly $β - α$ . The individual turns cancel; only their difference survives. (Algebraically: $R (α)^{⊤} R (β) = R (β - α)$ , because $R (α)^{⊤} = R (- α)$ and stacked rotations add.)

PICTURE. Same two arrows, both spun; the shaded gap between them is all the dot product feels.

Step 5 — Make the angle be the position

WHAT. Now we choose the angles. For a token at position $m$ , rotate its arrow by $α = m θ$ ; for position $n$ , rotate by $β = n θ$ . Here $θ$ (theta) is a fixed base frequency — radians of turn per step of position.

WHY. By Step 4 the score depends on $β - α = (n - m) θ$ . That $(n - m)$ is the relative distance between the two tokens. So the attention score now automatically reads "how far apart are these tokens?" — never their absolute slots. This is the promise the parent note made, now visible.

PICTURE. Positions $m = 1$ and $n = 4$ ; each step advances the arrow by the same $θ$ , and the gap is $(4 - 1) θ$ .

This is exactly the identity in the parent's Step 3 — now you saw why it holds.

Step 6 — Many frequencies: stacking flat worlds into $d$ dimensions

WHAT. Real heads have dimension $d$ (say 64). Split the vector into $d /2$ pairs. Pair $i$ (counting $i = 0, 1, \dots, \frac{d}{2} - 1$ ) lives in its own flat world and spins at its own frequency: $θ_{i} = 1000 0^{- 2 i / d} .$

HOW the pairs weave into one matrix. Each pair gets its own little $2 \times 2$ rotation $R (m θ_{i})$ , and we stack them down the diagonal of one big $d \times d$ matrix — a block-diagonal matrix, meaning every off-diagonal block is all zeros so the pairs never mix: $R_{m} = R (m θ_{0}) 0 ⋮ 0 0 R (m θ_{1}) 0 \dots \dots ⋱ \dots 00 ⋮ R (m θ_{d /2 - 1}) .$

Each $R (m θ_{i})$ ::: the $2 \times 2$ spin from Step 3, but with angle $m θ_{i}$ (position $m$ times that pair's speed).
the zeros ::: pair $i$ 's two coordinates are rotated only among themselves; dimension 1–2 never leak into dimension 3–4.

In code you never build this big matrix. You reshape $q$ into $d /2$ pairs, and for pair $i$ apply the same two lines from Step 3 with angle $m θ_{i}$ : $q_{2 i}^{'} = q_{2 i} cos (m θ_{i}) - q_{2 i + 1} sin (m θ_{i}), q_{2 i + 1}^{'} = q_{2 i} sin (m θ_{i}) + q_{2 i + 1} cos (m θ_{i}) .$ Do the same to $k$ with its position $n$ , then take the ordinary dot product $q^{' ⊤} k^{'}$ . Because the matrix is block-diagonal, that dot product is just the sum over pairs of each pair's rotated dot product — and Step 4 fires inside every pair, so each contributes a term depending only on $(n - m) θ_{i}$ . That is exactly how the $θ_{i}$ schedule gets woven into $Q K^{⊤}$ .

WHY (where the exponent comes from). We want the $d /2$ clocks to span a huge range of speeds, from "one radian per step" down to "barely turning at all", spaced evenly on a log scale (so each clock is a fixed ratio slower than the last — the natural way to cover many scales with few clocks). Demand two things:

The fastest clock ( $i = 0$ ) should turn at $θ_{0} = 1$ . Check: $1000 0^{- 0} = 1$ . ✓
The slowest clock ( $i = \frac{d}{2} - 1$ , essentially the largest $i$ ) should turn about $10000$ times slower, i.e. $θ \approx 1000 0^{- 1}$ .

The only smooth rule that hits $1$ at $i = 0$ and $\frac{1}{10000}$ at the top, evenly in log-space, is an exponential in $i$ : $θ_{i} = 1000 0^{- 2 i / d}$ . The exponent $- 2 i / d$ just re-labels the pair index $i$ (which runs $0 \dots \frac{d}{2} - 1$ ) onto the interval $[0, 1)$ so the base $10000$ is raised to a fraction from $0$ down to almost $- 1$ . The number $10000$ is a chosen constant fixing the ratio between the fastest and slowest clock. This is the same schedule used by Relative Position Bias (T5)-era sinusoidal encodings — RoPE reuses it as rotation speeds rather than additive waves.

WHY (what the range buys us). Fast clocks ( $θ$ large, small $i$ ) resolve nearby tokens sharply; slow clocks ( $θ$ small, large $i$ ) stay distinct over long ranges. Together they read distance at every scale — the mechanism behind the length-extrapolation gains discussed in Length Extrapolation in LLMs.

PICTURE. Three clocks side by side, spinning at different rates for the same position.

Step 7 — Edge and degenerate cases (the parts people skip)

WHAT / WHY / PICTURE, four short cases:

(a) Same position, $m = n$ . Then the relative distance is $0$ , so $(n - m) θ = 0$ and $R (0) = I$ (the identity — do nothing). The score is the plain, unrotated dot product $q^{⊤} k$ . Good: a token compared with itself carries no positional twist.

(b) The wrap-around. An angle only lives mod $2 π$ (a full circle brings an arrow back to where it started). Wrap-around happens when the total relative rotation satisfies $(n - m) θ \equiv 0 (mod 2 π)$ — note this is about $(n - m) θ$ , the whole accumulated gap, not the per-step angle $θ$ . Whenever two different relative distances land on angles $2 π$ apart, they produce the same arrow and hence the same score. This is why extrapolation works for a single clock: an absolute position never seen (say 2000) still produces a relative angle the model already met during training. (Multiple clocks at different $θ_{i}$ wrap at different distances, so together they stay unambiguous over the trained range.)

(c) Negative distance, $n < m$ . Then $(n - m) θ < 0$ : rotation goes clockwise instead of counter-clockwise. $R (- ϕ)$ is the inverse rotation — a reverse spin by $ϕ$ that exactly undoes $R (ϕ)$ (so $R (- ϕ) R (ϕ) = I$ ), not a reflection. Because the spin direction flips sign, "B before A" and "A before B" are cleanly distinguished by the sign of the angle — order is preserved.

(d) Values are never rotated. Attention computes $softmax (Q K^{⊤}) V$ , where $V$ is the value matrix — the stack of content vectors that actually get mixed together to form each token's output (the $Q, K$ only decide how much of each $V$ to take). RoPE spins only $q$ and $k$ ; the value vectors $V$ are left completely untouched. Position belongs in $Q K^{⊤}$ (which tokens attend to which) — twisting $V$ would corrupt the content being aggregated. This is a point worth stressing because the name "Rotary Embedding" tempts people to rotate everything, including the values; only the query–key pair carries the rotation.

PICTURE. The four cases on one board.

Worked check — parent Example 1, redrawn

$q = [1, 0]$ , $k = [0, 1]$ , $θ = π /4$ , positions $m = 0$ , $n = 3$ .

$R_{0} q = I q = [1, 0]$ (position 0 ⇒ angle 0 ⇒ no turn).
$R_{3} k = R (3 π /4) [0, 1] = [- sin \frac{3 π}{4}, cos \frac{3 π}{4}] = [- \frac{1}{2}, - \frac{1}{2}]$ .
Dot: $[1, 0] \cdot [- \frac{1}{2}, - \frac{1}{2}] = - \frac{1}{2} \approx - 0.707$ .

The one-picture summary

The whole derivation on one board: two arrows → spin each by its position-angle → the dot product forgets both absolute angles and keeps only the gap $(n - m) θ$ → stack many such flat worlds at frequencies $1000 0^{- 2 i / d}$ to read distance at every scale.

Recall Feynman retelling — say it back in plain words

Imagine every token is a little arrow. Instead of scribbling its position number next to it (the old "adding" way, which smears the arrow's meaning), we just spin the arrow — a bit for position 1, twice as much for position 2, and so on. Spinning never stretches the arrow, so its meaning is untouched; only its heading changes.

When we compare two tokens, we take a dot product, and a dot product is blind to where each arrow points on the page — it only feels the gap between them. Spin the first arrow by $m$ steps and the second by $n$ steps, and that gap becomes $(n - m)$ steps: pure relative distance. Absolute positions cancel out on their own.

To read distance at every scale, we don't use one arrow but many arrow-pairs, each spinning at its own speed — fast little clocks for nearby words, slow big clocks for far-apart words. Each pair sits in its own block of a big block-diagonal matrix, spinning independently and never mixing with the others. Because a full turn wraps back to the start, a position we never trained on still lands on a familiar relative angle, so the model keeps working on longer texts than it ever saw. And we only spin the queries and keys — the values, which carry the actual content we mix together, are left in peace.

Recall Quick self-test

Why does the attention score depend only on $n - m$ ? ::: Because a dot product only sees the angle between two arrows, and spinning $q$ by $m θ$ and $k$ by $n θ$ makes that between-angle equal $(n - m) θ$ ; the absolute turns cancel. Why lower dimensions get higher frequencies? ::: Fast clocks resolve nearby tokens but wrap too soon for long range; slow clocks stay distinct over long distances — so we assign fast to local (low $i$ ), slow to global (high $i$ ). When does wrap-around occur? ::: When the total relative rotation $(n - m) θ$ is a whole multiple of $2 π$ — i.e. $(n - m) θ \equiv 0 (mod 2 π)$ — not when the per-step angle $θ$ is zero. Why are values not rotated? ::: $V$ (the value/content vectors) is the stuff to aggregate; only $Q K^{⊤}$ decides which tokens to attend to, so position belongs there and not in $V$ .