Worked examples — Rotary positional embeddings (RoPE)

Everything here uses only the two objects from the parent note:

• a rotation matrix $R(\alpha )=\left[\begin{array}{cc}\cos \alpha & -\sin \alpha \\ \sin \alpha & \cos \alpha \end{array}\right]$ — "spin a 2D vector counter-clockwise by angle $\alpha$";
• the key identity $R(\alpha {)}^{T}R(\beta )=R(\beta -\alpha )$ — "undo one spin, then do the other, and only the gap $\beta -\alpha$ survives".

Two more symbols show up below, so let us pin them down now in plain words:

• $d$ = the head dimension — how many numbers are in one query or key vector for a single attention head (e.g. $d=64$). RoPE always chops this into $d/2$ independent pairs of coordinates.
• $i$ = the pair index, running $i=1,2,\dots ,d/2$. Pair $i$ is the $i$-th such 2-number block, and it gets its own spin-rate ${\theta }_i=10000^{-2i/d}$.

If the rotation matrix or the key identity feels shaky, re-read Rotary positional embeddings (RoPE) first, then come back.

---

The scenario matrix

Every RoPE calculation is one cell of this table. The examples below are labelled by cell. (Recall $d$ = head dimension, defined just above.)

Cell	Case class	What is unusual about it	Example
A	Same position, $m=n$	relative distance $0$ → no rotation gap	Ex 1
B	Positive gap, $n>m$	ordinary "key is after query"	Ex 2
C	Negative gap, $n<m$	angle gap flips sign → cosine unchanged, geometry mirrored	Ex 3
D	Degenerate vector, $q=0$ or $k=0$	rotation of the zero vector is still zero → score $0$	Ex 4
E	Wrap-around / limiting, huge $m$	angle $\mathrm{mod}2\pi$ — length extrapolation	Ex 5
F	Multi-frequency, $d>2$	two dimension-pairs spinning at different rates	Ex 6
G	Real-world word problem	translate a sentence into positions	Ex 7
H	Exam twist	"does rotating both by the SAME extra angle change the score?"	Ex 8

We check the score-only-depends-on-gap property everywhere, and along the way we cover both signs of the gap, zero, wrap-around, degeneracy, and multiple frequencies.

Recall What does

$R(\alpha {)}^{T}R(\beta )$ collapse to, and why do we care? It collapses to $R(\beta -\alpha )$ ::: because $R^T$ is the inverse rotation, so the two spins partly cancel and only the difference $\beta -\alpha$ remains — that difference is the relative position.

---

Example 1 — Cell A: same position ($m=n$)

Forecast: rotating both vectors by the same amount — does the dot product change at all compared to not rotating? Guess before reading.

1. Rotation angle for each. Both sit at position $2$, so each is spun by $m\theta =2\cdot \pi /6=\pi /3$. Why this step? Position $\times$ base angle is exactly how RoPE turns a position number into a rotation.

2. Use the key identity instead of rotating by hand. The score is $q^{\prime T}{k}^{\prime }=q^{T}R(m\theta {)}^{T}R(n\theta )k=q^{T}R(n\theta -m\theta )k=q^{T}R(0)k.$ Why this step? $R(0)=I$ (spin by nothing = do nothing), so we never touch the numbers.

3. Compute the plain dot product. $q^{T}k=3\cdot 3+4\cdot 4=25.$

Verify: Rotation is rigid — it preserves lengths and angles between vectors, so a shared rotation cannot change a dot product. Score $=25$, the same as the un-rotated dot product. When two tokens share a position, RoPE is invisible. ✅

---

Example 2 — Cell B: positive gap ($n>m$)

The geometry of this example is drawn in Figure s01: the cyan arrow is the rotated query $q^{\prime }$ at position 1, the amber arrow is the rotated key $k^{\prime }$ at position 4, and the white arc between them is the effective gap angle $3\theta =135^{\circ }$. Use the picture to see why the score comes out negative before you crunch the numbers.

Forecast: the gap is $n-m=3$, so the effective rotation is $3\theta =3\pi /4=135^{\circ }$. Will the score be positive or negative? Look at Figure s01 and guess from the angle between the cyan and amber arrows.

1. Effective gap angle. $(n-m)\theta =3\cdot \pi /4=3\pi /4$. Why this step? Only the difference matters, so we skip the two separate spins.

2. Collapse to one rotation. $\text{score}=q^{T}R(3\pi /4)k.$ Why this step? Key identity again — $R(1\cdot \theta {)}^{T}R(4\cdot \theta )=R(3\theta )$.

3. Multiply out. With $\cos (3\pi /4)=-\frac{1}{\sqrt{2}}$, $\sin (3\pi /4)=\frac{1}{\sqrt{2}}$, $R(3\pi /4)k=\left[\begin{array}{cc}-\frac{1}{\sqrt{2}}& -\frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}}& -\frac{1}{\sqrt{2}}\end{array}\right]\left[\begin{array}{c}0\\ 1\end{array}\right]=\left[\begin{array}{c}-\frac{1}{\sqrt{2}}\\ -\frac{1}{\sqrt{2}}\end{array}\right].$

4. Dot with $q$. $q^{T}R(3\pi /4)k=[1,0]\cdot \left[\begin{array}{c}-1/\sqrt{2}\\ -1/\sqrt{2}\end{array}\right]=-\frac{1}{\sqrt{2}}\approx -\mathrm{0.707.}$

Verify: In Figure s01 the two arrows point more than $90^{\circ }$ apart, so their dot product must be negative — matches $-0.707$. ✅ (This mirrors Example 1 in the parent note, which used positions $0$ and $3$; same gap of $3$, same answer.)

---

Example 3 — Cell C: negative gap ($n<m$)

Forecast: the gap flipped sign. Will the score match Example 2, or flip sign too?

1. Gap angle. $(n-m)\theta =-3\cdot \pi /4=-3\pi /4$. Why this step? The formula is $q^{T}R((n-m)\theta )k$ — negative gap → negative angle.

2. Apply $R(-3\pi /4)$. Here $\cos (-3\pi /4)=-\frac{1}{\sqrt{2}}$ (even) but $\sin (-3\pi /4)=-\frac{1}{\sqrt{2}}$ (odd): $R(-3\pi /4)k=\left[\begin{array}{cc}-\frac{1}{\sqrt{2}}& \frac{1}{\sqrt{2}}\\ -\frac{1}{\sqrt{2}}& -\frac{1}{\sqrt{2}}\end{array}\right]\left[\begin{array}{c}0\\ 1\end{array}\right]=\left[\begin{array}{c}\frac{1}{\sqrt{2}}\\ -\frac{1}{\sqrt{2}}\end{array}\right].$ Why this step? We must track the sign flip in $\sin$ carefully — this is exactly where careless students drop a minus.

3. Dot with $q$. $[1,0]\cdot \left[\begin{array}{c}1/\sqrt{2}\\ -1/\sqrt{2}\end{array}\right]=+\frac{1}{\sqrt{2}}\approx +\mathrm{0.707.}$

Verify: The score is $+0.707$, the negative of Example 2's $-0.707$. That is not a coincidence: swapping $q$ and $k$ here swaps which vector was $[1,0]$ vs $[0,1]$, and for these particular perpendicular vectors the score is antisymmetric. The magnitude $|-0.707|=|0.707|$ is identical because $\cos$ of $+3\pi /4$ and $-3\pi /4$ agree — the geometry is a mirror image. ✅

---

Example 4 — Cell D: degenerate vector ($k=0$)

Forecast: does the position machinery still do anything?

1. Rotate the zero vector. $R(\alpha )[0,0{]}^T=[0,0{]}^T$ for any $\alpha$. Why this step? A rotation is a linear map; it sends the origin to the origin no matter the angle.

2. Dot product. $q^{\prime T}{k}^{\prime }=q^{\prime T}\cdot [0,0{]}^T=0$.

Verify: No rotation, no frequency, no position could ever make this non-zero — a token with a zero key contributes zero to the pre-softmax score. Units check: a dot product of vectors is a pure number, and $0$ is dimensionless. ✅ This is the degenerate boundary of the whole method.

---

Example 5 — Cell E: wrap-around & length extrapolation

Forecast: position 2000 was never seen in training (max length 512). Guess whether RoPE breaks.

1. Compute the base frequency. ${\theta }_1=10000^{-1/32}=e^{-\frac{\ln 10000}{32}}\approx \mathrm{0.7499.}$ Why this step? This is the actual spin-rate of the first (fastest) dimension pair.

2. Accumulated raw angle. $m{\theta }_1=2000\times 0.7499\approx 1499.8$ radians. Why this step? Position $\times$ frequency, exactly as in the parent note.

3. Wrap into $[0,2\pi )$. Divide by $2\pi \approx 6.2832$: $1499.8/6.2832\approx 238.7$ full turns. Subtract the $238$ whole turns ($238\times 2\pi \approx 1495.4$): $1499.8-1495.4\approx 4.40$ radians. Why this step? Rotation is $2\pi$-periodic: spinning $1499.8$ rad lands in the same place as spinning $4.40$ rad. The absolute number is huge but the geometry is ordinary.

4. The gap that actually matters. For positions $2000$ and $2005$, the gap angle is $(2005-2000){\theta }_1=5\times 0.7499\approx 3.75\text{radians}.$ Why this step? Attention only uses the gap. A relative distance of $5$ was seen thousands of times in training (any pair 5 apart), so this angle is completely familiar.

Verify: $0\le 4.40<2\pi \approx 6.28$ ✅ and the gap $3.75$ radians is independent of the absolute positions — the model reuses knowledge it already has. This is precisely why RoPE extrapolates to unseen lengths. (Matches Example 2 in the parent note.) ✅

---

Example 6 — Cell F: multiple frequencies ($d=4$)

Figure s02 shows the pay-off of splitting a vector into pairs: for the same relative distance, the cyan "fast" pair (spin-rate $1.0$) sweeps a wide $1.0$-radian arc, while the amber "slow" pair (spin-rate $0.5$) barely turns. That is why one head can be sensitive to short and long ranges at once.

Forecast: two pairs spin at different speeds. Does the total score just add the two pair-scores?

1. Split into pairs. Pair 1 = $[1,0]$, pair 2 = $[1,0]$. Each pair is rotated independently — that is exactly the block-diagonal structure defined just above (see Figure s02). Why this step? RoPE never mixes dimension-pairs; each is its own little 2D rotation.

2. Gap angle per pair. Gap $=n-m=1$, so

   • pair 1: $1\cdot {\theta }_1=1.0$ rad
   • pair 2: $1\cdot {\theta }_2=0.5$ rad.

3. Score of each pair $=q^{T}R(\text{gap}\cdot \theta )k$. For $q=k=[1,0]$: $[1,0]R(\alpha )[1,0{]}^T=\cos \alpha .$ Why this step? $R(\alpha )[1,0{]}^T=[\cos \alpha ,\sin \alpha {]}^T$, and dotting with $[1,0]$ picks out $\cos \alpha$.

   • pair 1 score $=\cos (1.0)\approx 0.5403$
   • pair 2 score $=\cos (0.5)\approx 0.8776$.

4. Total score = sum of pair scores (a dot product over $d=4$ is the sum of the two 2D dot products): $0.5403+0.8776\approx \mathrm{1.4180.}$

Verify: Figure s02 shows the fast pair (cyan) swept $1.0$ rad and the slow pair (amber) swept only $0.5$ rad for the same relative distance — the slow pair stays closer to its start, so its cosine ($0.88$) is larger than the fast pair's ($0.54$). Sum $\approx 1.42$. Different frequencies = different sensitivity to distance. ✅

---

Example 7 — Cell G: word problem

Forecast: guess the angle before computing, and whether absolute positions matter.

1. Identify query & key positions. Query "ran" $\Rightarrow m=8$; key "cat" $\Rightarrow n=2$. Why this step? By convention the token doing the looking is the query.

2. Relative position. $n-m=2-8=-6$ (the key is 6 tokens before the query).

3. Gap angle. $(n-m)\theta =-6\times 0.2=-1.2$ rad. Why this step? Attention score uses $R((n-m)\theta )$; sign tells us direction (key before query → negative).

Verify: Any subject–verb pair exactly 6 apart — say positions $(50,44)$ — gives $(44-50)\times 0.2=-1.2$ rad, identical. RoPE lets "ran" recognise its subject by the pattern of the gap, not by where the clause happens to sit in the sentence. Units: radians, an angle, as required. ✅

---

Example 8 — Cell H: exam twist (global shift invariance)

Forecast: does adding 100 to both positions change anything?

1. Original gap. $n-m=5-2=3$. Score $=q^{T}R(3\theta )k$.

2. Shifted gap. New positions $102$ and $105$: gap $=(105)-(102)=3$. Unchanged. Why this step? The constant $c$ appears in both, so it cancels in the difference — this is the heart of "relative, not absolute".

3. Numerical score. $3\theta =\pi$, so $R(\pi )=\left[\begin{array}{cc}-1& 0\\ 0& -1\end{array}\right]=-I$. Then $q^{T}R(\pi )k=q^T(-k)=-(2\cdot 1+1\cdot (-1))=-(2-1)=-1.$

Verify: Both the original and shifted setups give score $-1$, because $R(\pi )$ is $-I$ regardless of $c$. Global position shifts are invisible to RoPE — exactly the property that distinguishes it from Sinusoidal Positional Encoding additions and lets it match ideas like Relative Position Bias (T5) and ALiBi (Attention with Linear Biases). ✅

---

Wrap-up

Recall Which two vectors get rotated, and which does not?

Queries and keys are rotated ::: values are left untouched, because position must shape which tokens to attend to, not what content to aggregate.

Recall Why does the score depend only on

$n-m$? Because $R(m\theta {)}^{T}R(n\theta )=R((n-m)\theta )$ ::: the two absolute spins partially cancel and only their difference — the relative position — survives.

See also Multi-Head Attention (each head can rotate independently), Attention Mechanism (where the rotated $Q{K}^T$ lives), and Length Extrapolation in LLMs (Example 5's real payoff). The Hinglish walkthrough is at 4.1.07 Rotary positional embeddings (RoPE) (Hinglish).