Question bank — Rotary positional embeddings (RoPE)

False. Vanilla sinusoidal encoding adds; RoPE rotates the query and key vectors instead, so position never contaminates the raw content by addition.

False. $R(\theta )$ is orthogonal ($R^{T}R=I$), so it preserves length — it only swings the direction. This is exactly why a token's "size" (norm) survives RoPE untouched.

False. Only queries and keys are rotated. Values carry content to be aggregated; rotating them would twist the payload, not just steer the attention weights.

True. The dot product collapses to $q^{T}R((n-m)\theta )k$, which depends only on the difference $n-m$, so absolute placement drops out.

False. It splits $d$ into $d/2$ independent pairs and spins each pair by its own frequency ${\theta }_i=10000^{-2i/d}$, giving a spread of fast and slow rotations.

False. $d$ (the head dimension $d_k$) must be even, because rotation acts on 2D planes and the coordinates are paired as $(x_1,x_2),(x_3,x_4),\dots$; an odd $d$ would leave one coordinate with no partner to rotate against.

False. Lower dimensions (small $i$) use higher frequencies (fast rotation, fine local detail); higher dimensions (large $i$) use lower frequencies (slow rotation, long-range structure).

Only partly. Attention reads relative gaps, and a gap seen in training keeps the same angle at any absolute position — so short gaps extrapolate cleanly. But it is not free: each frequency ${\theta }_i$ has its own wrap period $2\pi /{\theta }_i$, the slow high-$i$ pairs sweep angles at very long range that were never seen during training, so extrapolation degrades there — the qualification in Length Extrapolation in LLMs.

False. That is Relative Position Bias (T5). RoPE bakes relative position into the geometry of the dot product — no extra learned parameters.

False. In Multi-Head Attention each head applies RoPE to its own $q,k$, so heads can specialise to different position scales independently.

False. It modifies the $q^{T}k$ score inside the Attention Mechanism before softmax; the softmax and value aggregation are unchanged.

Wrong sign. The transpose reverses the first spin, $R(\alpha {)}^T=R(-\alpha )$, so $R(\alpha {)}^{T}R(\beta )=R(\beta -\alpha )$, giving the difference $n-m$, not the sum — that difference is the entire point.

Error. Attention $=\text{softmax}(q^{T}k)v$; position belongs in which tokens to fetch ($q^{T}k$), not in what content ($v$) gets fetched. Rotating $v$ would corrupt the payload.

Error. Higher frequency = faster spin = fine short-range detail. Long-range structure needs slow spin (low frequency), which is precisely why high dimensions get ${\theta }_i=10000^{-2i/d}$ shrinking toward zero.

Error. Rotation lives in a 2D plane, so you must pair coordinates: $(x_1,x_2)$ spin together, $(x_3,x_4)$ together. A lone number cannot be "rotated" — this is also why $d$ must be even.

Error. The $1/\sqrt{d_k}$ scaling controls dot-product magnitude and is independent of position. RoPE leaves it in place.

Error. Learned embeddings fail because absolute slot 2000 has no learned vector; RoPE has no per-position parameters — it computes the angle, and attention only reads the relative gap, which was trained.

Because the dot product of two rotated vectors depends only on the angle difference ($R(\alpha {)}^{T}R(\beta )=R(\beta -\alpha )$). No other cheap operation makes relative position "fall out" of the attention score for free.

The transpose in $q^{T}R(m\theta {)}^{T}R(n\theta )k$ reverses the first rotation into $R(-m\theta )$; composing rotations adds turns, so the reversed $-m\theta$ against the forward $+n\theta$ leaves $R((n-m)\theta )$ — absolute offsets cancel and only the gap $n-m$ survives.

A single frequency ${\theta }_i$ aliases: two positions $2\pi /{\theta }_i$ apart give the identical angle, so that pair alone cannot tell them apart. Stacking $d/2$ different frequencies means the full set of angles almost never collides at once — the pair that aliases for one distance stays distinct in every other pair.

Position should steer attention weights, not distort the information being summed. $v$ is the information; spinning it would answer "where" by damaging "what."

It reuses a geometric spread of wavelengths from short to very long, giving a smooth ladder of scales — a proven way to cover many distances without gaps.

ALiBi adds a fixed penalty $-\lambda |m-n|$ straight onto each attention score (closer tokens get less penalty, farther more), a purely additive bias with no rotation. RoPE instead rotates $q$ and $k$ so relative distance emerges from the dot-product geometry — same goal, different mechanism.

The identity — $R(0)=I$ leaves the vector exactly as-is, so the first token carries no rotation, only pure content.

Zero. $R((n-m)\theta )=R(0)=I$, so self-attention scores feel no position twist at all — as they should, since the gap is zero.

Yes. $R((n-m)\theta )$ is just a rotation by a negative angle (spin the other way); the geometry is symmetric and well-defined for any signed gap.

It still rotates normally: writing the rotation angle $m{\theta }_i$, the pair becomes $[x_1\cos (m{\theta }_i),x_1\sin (m{\theta }_i)]$. A zero component is a valid starting point, not a degenerate case — the pair simply starts on an axis.

That pair's rotation aliases to the same angle, so it cannot distinguish those two distances — but the other frequencies still differ, so the full vector keeps them apart. This is why multiple frequencies matter.

Yes, and that is intended. Fast wrapping still yields consistent relative angle differences for a given gap, so training on those gaps transfers to long sequences — the basis of RoPE extrapolation.

The rotation $m{\theta }_i$ becomes tiny, so those pairs barely turn even across a long sequence. Two far-apart positions then produce almost-identical angles there, and in finite floating-point precision the difference can round away, weakening long-range discrimination in exactly the pairs meant to handle it.