Exercises — Rotary positional embeddings (RoPE)

(b) — only queries and keys. The attention score $q^{T}k$ is where position needs to live, because it decides which tokens attend to which. Values carry content to be summed and are left untouched. Answer: (b).

$R(\theta )=\left[\begin{array}{cc}\cos \theta & -\sin \theta \\ \sin \theta & \cos \theta \end{array}\right],R(0)=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]=I.$ A zero-angle turn does nothing: $\cos 0=1$, $\sin 0=0$. So a token at position 0 keeps its query unchanged. This is the natural "anchor" of the sequence.

Fast. With $i$ small, the exponent $-2i/d$ is close to $0$, so ${\theta }_i$ is close to $10000^0=1$ — a large angle per step. Low dimension pairs spin quickly (fine, local detail); high dimension pairs spin slowly (coarse, long-range detail). See the parent note's Mistake 3.

Since $m=0$, $q^{\prime }=R(0)q=q=(1,0)$. The key rotates by $3\theta =3\pi /4=135^{\circ }$. Then $\cos (3\pi /4)=-\frac{1}{\sqrt{2}}$, $\sin (3\pi /4)=+\frac{1}{\sqrt{2}}$: $k^{\prime }=\left[\begin{array}{cc}-\frac{1}{\sqrt{2}}& -\frac{1}{\sqrt{2}}\\ \frac{1}{\sqrt{2}}& -\frac{1}{\sqrt{2}}\end{array}\right]\left[\begin{array}{c}0\\ 1\end{array}\right]=\left[\begin{array}{c}-\frac{1}{\sqrt{2}}\\ -\frac{1}{\sqrt{2}}\end{array}\right].$ Wait — check the sign. Column 2 of $R(3\pi /4)$ is $(-\sin ,\cos )=(-\frac{1}{\sqrt{2}},-\frac{1}{\sqrt{2}})$. Multiplying by the $1$ in the second slot gives $k^{\prime }=(-\frac{1}{\sqrt{2}},-\frac{1}{\sqrt{2}})$. Score $=q^{\prime T}{k}^{\prime }=(1)(-\frac{1}{\sqrt{2}})+(0)(-\frac{1}{\sqrt{2}})=-\frac{1}{\sqrt{2}}\approx -0.707$. Answer: $-\frac{1}{\sqrt{2}}$.

(The parent note wrote $-1/\sqrt{2}$ for the same setup — matches, once the second column's signs are read correctly.)

The score is $q^{T}R(m\theta {)}^{T}R(n\theta )k=q^{T}R((n-m)\theta )k$. Relative distance is $n-m=4-1=3$, exactly as in 2.1 where $3-0=3$. So the angle inside is $3\theta =3\pi /4$ both times. Concretely: $R((n-m)\theta )k=R(3\pi /4)(0,1)=(-\frac{1}{\sqrt{2}},-\frac{1}{\sqrt{2}})$, and $q^T$ of that is $-\frac{1}{\sqrt{2}}$. Answer: $-\frac{1}{\sqrt{2}}$, unchanged — this is the whole point of RoPE: only the gap $n-m$ matters, not the absolute slots.

${\theta }_1=10000^{-2\cdot 1/64}=10000^{-1/32}.$ Take logs: $-\frac{1}{32}\ln 10000=-\frac{1}{32}(9.2103)=-0.28782$, so ${\theta }_1=e^{-0.28782}\approx 0.7499$. Accumulated angle at $m=10$: $10\times 0.7499\approx 7.499$ radians. Since $7.499-2\pi \approx 1.216$, that is just past one full turn. Answer: ${\theta }_1\approx 0.750$, angle at position 10 $\approx 7.50$ rad ($\equiv 1.22$ rad mod $2\pi$).

Let $x=(x_1,x_2)$ and $y=R(\theta )x=(x_1\cos \theta -x_2\sin \theta ,x_1\sin \theta +x_2\cos \theta )$. $\Vert y{\Vert }^2=(x_1\cos \theta -x_2\sin \theta {)}^2+(x_1\sin \theta +x_2\cos \theta {)}^2.$ Expand: the cross terms $-2x_1{x}_2\cos \theta \sin \theta$ and $+2x_1{x}_2\sin \theta \cos \theta$ cancel. What remains is $x_1^2({\cos }^2\theta +{\sin }^2\theta )+x_2^2({\sin }^2\theta +{\cos }^2\theta )=x_1^2+x_2^2=\Vert x{\Vert }^2,$ using ${\cos }^2\theta +{\sin }^2\theta =1$. Why it matters: RoPE only turns the query/key arrows, never stretches them. So it injects position without secretly boosting or shrinking the content magnitude — the attention score changes only through the angle, exactly as intended.

One full turn happens when $P{\theta }_i=2\pi$, so $P=2\pi /{\theta }_i$.

• $i=1$: ${\theta }_1\approx 0.7499$, so $P_1=2\pi /0.7499\approx 8.38$ positions. This pair completes a loop roughly every 8 tokens — it tracks local structure.
• $i=32$: ${\theta }_{32}=10000^{-64/64}=10000^{-1}=10^{-4}$, so $P_{32}=2\pi /10^{-4}\approx 62,832$ positions. This pair barely turns across an entire long document — it tracks global structure. Interpretation: the head simultaneously holds a fast "second hand" (pair 1) and a slow "hour hand" (pair 32), covering all scales at once.

$R(g)k=(\cos g,\sin g)$, and $q^T$ of that picks the first component: score $=\cos (g)$ — a smooth curve defined for every real $g$. Continuous landmarks (imagine sliding $g$ continuously):

• Maximum $=+1$ at $g=0$ (a token attending to itself, arrows aligned).
• First zero at $g=\pi /2\approx 1.571$ (arrows perpendicular).
• Minimum $=-1$ at $g=\pi \approx 3.14$ (arrows opposed), then it climbs back — the score oscillates and wraps.

Integer gaps (what a real sentence can actually have, since positions are whole numbers): $g=1\Rightarrow \cos 1\approx 0.540$, $g=2\Rightarrow \cos 2\approx -0.416$, $g=3\Rightarrow \cos 3\approx -0.990$. So real gaps just sample points off the smooth cosine curve; the continuous zero at $g=\pi /2$ falls between integer gaps 1 and 2, which is why no integer gap gives exactly zero here. Answer: score $=\cos g$; continuous max at $g=0$, zero at $g=\pi /2$, min at $g=\pi$; the reachable integer gaps sample that curve.

Step A — transpose is reverse rotation. From the preliminaries, $R(\alpha {)}^T=R(-\alpha )$. What we just did: recognised that un-doing a rotation is rotating backwards. Step B — multiply out one entry to see the pattern. Write $R(-\alpha )R(\beta )$ and compute, say, the top-left entry. Row 1 of $R(-\alpha )$ is $[\cos \alpha ,\sin \alpha ]$ (since $\cos (-\alpha )=\cos \alpha$, $-\sin (-\alpha )=\sin \alpha$); column 1 of $R(\beta )$ is $[\cos \beta ,\sin \beta {]}^T$. Their product is $\cos \alpha \cos \beta +\sin \alpha \sin \beta =\cos (\beta -\alpha ),$ which is precisely the angle-difference identity — and precisely the top-left entry of $R(\beta -\alpha )$. Do the same for the other three entries (top-right gives $-\sin (\beta -\alpha )$, etc.) and every slot lines up, so $R(-\alpha )R(\beta )=R(\beta -\alpha ).$ Step C — apply to attention. With $q^{\prime }=R(m\theta )q$, $k^{\prime }=R(n\theta )k$: $q^{\prime T}{k}^{\prime }=q^{T}R(m\theta {)}^{T}R(n\theta )k=q^{T}R(n\theta -m\theta )k=q^{T}R((n-m)\theta )k.$ The $n<m$ case: if the key sits before the query, then $g=n-m$ is negative. Nothing breaks — the identity holds for any real angles, so we simply rotate by a negative angle, and $R(-|g|\theta )=R(|g|\theta {)}^T$. Geometrically the relative arrow just turns the other way. The score becomes $q^{T}R(-|g|\theta )k$, giving $\cos (|g|\theta )$ for the $q=k=(1,0)$ case (because $\cos$ is even: $\cos (-x)=\cos x$) — so the raw score is symmetric in the two directions, while the sign of the rotation still distinguishes "before" from "after" in the off-diagonal terms. $■$

Euler's formula: $e^{im\theta }=\cos (m\theta )+i\sin (m\theta )$. Multiply: $z{e}^{im\theta }=(x_1+i{x}_2)(\cos m\theta +i\sin m\theta ).$ Expand, remembering $i^2=-1$: $=\underset{\text{real part}}{\underbrace{(x_1\cos m\theta -x_2\sin m\theta )}}+i\underset{\text{imaginary part}}{\underbrace{(x_1\sin m\theta +x_2\cos m\theta )}}.$ The real part and imaginary part are exactly the two rows of $R(m\theta )(x_1,x_2)$ from the definition box. So one complex multiply = one 2D rotation. Why libraries do this: it is one multiply instead of a $2\times 2$ matrix product, and it vectorises cleanly across all $d/2$ pairs. This is the "complex number trick" in the parent note.

$1500\mathrm{mod}2\pi$: divide $1500/(2\pi )=1500/6.28319\approx 238.732$, so $238$ full turns use up $238\times 2\pi =1495.575$ radians. Remainder $=1500-1495.575\approx 4.42$ radians. (The parent note's $\approx 1.86$ used a heavily rounded ${\theta }_1$; with ${\theta }_1=0.75$ exactly the remainder is $\approx 4.42$ — same idea either way: it lands somewhere on the circle we have already visited.) Why extrapolation survives: the score depends on the relative angle $g{\theta }_1$. A token pair at positions $(2000,2005)$ has gap $g=5$, giving $5\times 0.75=3.75$ rad — a relative angle the model met countless times while training on short sequences. Absolute position 2000 is brand new, but its gap-angles are old friends, because rotations wrap around every $2\pi$. So the rotation matrices generalise to unseen absolute positions as long as the relative distances were seen. See Length Extrapolation in LLMs. Answer: $1500\mathrm{mod}2\pi \approx 4.42$ rad.

Attention output for query $m$ is ${\sum }_n{\text{softmax}}_n(q_m^{\prime T}{k}_n^{\prime })v_n$. If we also rotate values to $v_n^{\prime }=R(n\theta )v_n$, then the summed output carries a jumble of absolute rotations ${\sum }_n{a}_{n}R(n\theta )v_n$ that does not reduce to a clean relative form — different $n$'s get different absolute turns and never cancel, because there is no second transpose to meet them. The content vector delivered to the next layer would now spin with absolute position, re-introducing exactly the absolute-position bias RoPE set out to kill. Conclusion: rotating $V$ breaks relative invariance. Keep $V$ untouched. Contrast with Relative Position Bias (T5), which also keeps values clean and only edits the score.

RoPE is block-diagonal: pair 1 (dims 1–2) rotates by $g{\theta }_1$, pair 2 (dims 3–4) by $g{\theta }_2$, and the 4D dot product is the sum of the two 2D dot products.

• Pair 1: $q_{(1,2)}=(1,0)$, $k_{(1,2)}=(1,0)$. Score $=q^{T}R(g\cdot 1)k=\cos (g)$ (from Exercise 3.3).
• Pair 2: $q_{(3,4)}=(1,0)$, $k_{(3,4)}=(1,0)$. Score $=\cos (0.1g)$. Total $=\cos (g)+\cos (0.1g)$. At $g=3$: $\cos 3+\cos 0.3=(-0.98999)+(0.95534)=-0.03465$. Answer: total score at $g=3$ is $\approx -0.0347$. This confirms the multi-frequency, block-diagonal picture: each pair contributes an independent cosine, summed. Different heads in Multi-Head Attention can lean on different pairs.

Each pair contributes $\cos (g{\theta }_i)$. At a large gap $g$, the fast pairs (${\theta }_i$ near 1) have argument $g{\theta }_i$ sweeping through many periods; across the $d/2$ pairs these cosines land at effectively random phases and partly cancel when summed, so the aggregate score shrinks toward its average — an emergent decay, not a designed one (this is the correct reading of the L3 trap). But the slow pairs (${\theta }_i\approx 10^{-4}$) barely move even at $g$ in the thousands, so they cannot represent a new large relative distance distinctly — the model has no trained signal for "gap = 3000" if it only ever saw gaps up to 512. Fix in practice: position-interpolation / NTK scaling stretches or re-bases the frequencies so that unseen large gaps map back into the trained angular range, restoring usable relative signal. See Length Extrapolation in LLMs. Answer (conceptual): decay is an emergent phase-cancellation effect; extreme extrapolation still needs frequency rescaling because slow pairs run out of unseen angular resolution.

RoPE rotates only the queries and keys, never the values ::: because the score $q^{T}k$ is where position must live, while values carry content. The identity that collapses two positions into their gap is ::: $R(\alpha {)}^{T}R(\beta )=R(\beta -\alpha )$. Low dimension pairs use high frequencies for ::: local/fine structure; high pairs use low frequencies for long-range structure. A single RoPE pair's score as a function of gap $g$ is ::: $\cos (g\theta )$ — it oscillates, it does not decay.