Exercises — Positional encodings (sinusoidal)

The rule: even index → sine, odd index → cosine. Index parity, not the pair index $i$, decides.

• Dim $0$ (even) → $\sin$, pair $i=0$
• Dim $1$ (odd) → $\cos$, pair $i=0$
• Dim $2$ (even) → $\sin$, pair $i=1$
• Dim $3$ (odd) → $\cos$, pair $i=1$
• Dim $4$ (even) → $\sin$, pair $i=2$
• Dim $5$ (odd) → $\cos$, pair $i=2$

Three pairs, each a $(\sin ,\cos )$ couple sharing one frequency ${\omega }_i$. That shared frequency is what lets the pair trace a point on a circle — the geometry that gives us the rotation ${\mathbf{T}}_k$ defined above.

${\omega }_1=\frac{1}{10000^{2\cdot 1/4}}=\frac{1}{10000^{1/2}}=\frac{1}{100}=0.01$ ${\lambda }_1=\frac{2\pi }{{\omega }_1}=2\pi \cdot 100=200\pi \approx 628.3\text{ positions}$ This slow wave takes ~628 positions to repeat — it encodes coarse position.

Pairs: $i\in \{0,1\}$. Frequencies straight from the definition ${\omega }_i=1/10000^{2i/d_{model}}$: ${\omega }_0=\frac{1}{10000^{0/4}}=\frac{1}{10000^0}=\frac{1}{1}=1,{\omega }_1=\frac{1}{10000^{2/4}}=\frac{1}{100}=0.01$

• $P{E}_{(2,0)}=\sin ({\omega }_0\cdot 2)=\sin (2)=0.9093$
• $P{E}_{(2,1)}=\cos ({\omega }_0\cdot 2)=\cos (2)=-0.4161$
• $P{E}_{(2,2)}=\sin ({\omega }_1\cdot 2)=\sin (0.02)=0.0200$
• $P{E}_{(2,3)}=\cos ({\omega }_1\cdot 2)=\cos (0.02)=0.9998$

$PE(2)=[0.9093,-0.4161,0.0200,0.9998]$ Why 4 decimals is safe here: $\sin ,\cos$ never exceed magnitude $1$ and their slope is at most $1$, so a rounding error of $10^{-5}$ in the angle can move the output by at most $10^{-5}$ — well inside the 4th decimal. The values are stable to report.

At $pos=0$ every sine/cosine argument is ${\omega }_i\cdot 0=0$:

• Even dims: $\sin (0)=0$
• Odd dims: $\cos (0)=1$

So $PE(0)=[0,1,0,1,0,1,\dots ]$ — an alternating zero/one pattern, independent of $d_{model}$'s frequencies. Geometrically every pair starts at angle $0$, i.e. pointing along the positive-$\cos$ axis — this is the yellow "start" dot at $(1,0)$ in figure s02.

Add $PE(2)=[0.9093,-0.4161,0.0200,0.9998]$ component-wise: $[0.7+0.9093,0.1-0.4161,-0.4+0.0200,0.3+0.9998]=[1.6093,-0.3161,-0.3800,1.2998]$

Using the shift matrix ${\mathbf{T}}_k$ defined at the top of this page with $k=1$ and ${\omega }_1=0.01$: ${\mathbf{T}}_1=\left[\begin{array}{cc}\cos ({\omega }_1)& \sin ({\omega }_1)\\ -\sin ({\omega }_1)& \cos ({\omega }_1)\end{array}\right]=\left[\begin{array}{cc}\cos (0.01)& \sin (0.01)\\ -\sin (0.01)& \cos (0.01)\end{array}\right]\approx \left[\begin{array}{cc}0.99995& 0.0100\\ -0.0100& 0.99995\end{array}\right]$ Why exactly this form: moving one step adds ${\omega }_1$ to the pair's angle (see the definition callout and figure s03), and rotating a circle-point by a fixed angle is by definition multiplication by $R({\omega }_1)$. Why determinant $1$: $\det ={\cos }^2({\omega }_1)+{\sin }^2({\omega }_1)=1$ by the Pythagorean identity — a pure rotation neither stretches nor shrinks, so no information about magnitude is lost, only the angle turns.

$\left[\begin{array}{cc}0.99995& 0.0100\\ -0.0100& 0.99995\end{array}\right]\left[\begin{array}{c}0.0200\\ 0.9998\end{array}\right]$ Top: $0.99995(0.0200)+0.0100(0.9998)=0.0200+0.0100=0.0300$ Bottom: $-0.0100(0.0200)+0.99995(0.9998)=-0.0002+0.9998=0.9996$ Direct compute: $P{E}_{(3,2)}=\sin (0.03)=0.0300$, $P{E}_{(3,3)}=\cos (0.03)=0.9996$. ✅ They match — the rotation IS the position shift.

Recall the pairing: dimension $0$ lives in pair $i=0$ with ${\omega }_0=1$; dimension $2$ lives in pair $i=1$ with ${\omega }_1=0.01$.

• Dim 0: $\Delta =\sin ({\omega }_0\cdot 1)-\sin ({\omega }_0\cdot 0)=\sin (1)-\sin (0)=0.8415$
• Dim 2: $\Delta =\sin ({\omega }_1\cdot 1)-\sin ({\omega }_1\cdot 0)=\sin (0.01)-\sin (0)=0.0100$

Dimension 0 changes ~84× more. Reason: ${\omega }_0=1$ is the highest frequency (fast wiggle), so a one-step move sweeps a big angle; ${\omega }_1=0.01$ barely moves. High-frequency dims track fine local order; low-frequency dims track coarse global position — exactly the "zoom levels" the parent note describes.

${\lambda }_1=2\pi \cdot B^{2i/d_{model}}=2\pi \cdot B^{2/4}=2\pi \sqrt{B}$ Set equal to $1000$: $2\pi \sqrt{B}=1000\Rightarrow \sqrt{B}=\frac{1000}{2\pi }=159.155\Rightarrow B=159.155^2\approx 25330$ So a base of roughly $B\approx 25330$ stretches the longest wavelength to $\sim 1000$ positions. Bigger base → longer wavelengths → better long-range coverage but coarser resolution.

$\sin (0.01\cdot pos)=0$ with the same upward slope repeats after one full period ${\lambda }_1=2\pi /0.01=628.318$ positions. Since $pos$ must be an integer, the nearest is $pos=628$: $\sin (0.01\cdot 628)=\sin (6.28)\approx -0.00319\approx 0$ This shows why ONE frequency is ambiguous — position $0$ and $\sim 628$ look alike in that single dimension. The other dimensions (with different frequencies) break the tie, giving every position a unique full vector.

The pair's two components at position $pos$ form the vector $(\sin ({\omega }_{i}pos),\cos ({\omega }_{i}pos))$ (same $pos$ we've used all page — no new letters). Dot the vectors at $po{s}_1$ and $po{s}_2$: $\sin ({\omega }_{i}po{s}_1)\sin ({\omega }_{i}po{s}_2)+\cos ({\omega }_{i}po{s}_1)\cos ({\omega }_{i}po{s}_2)=\cos ({\omega }_i(po{s}_1-po{s}_2))$ using $\cos (A-B)=\cos A\cos B+\sin A\sin B$. The result is $\cos ({\omega }_i\cdot \Delta )$ where $\Delta =po{s}_1-po{s}_2$ — absolute positions cancelled; only the gap survives. This is the seed of relative attention: similarity between positions is a clean function of their distance. Numeric: $\Delta =5-8=-3$, $\cos (0.01\cdot (-3))=\cos (-0.03)=\cos (0.03)=0.99955$.

The angle is ${\omega }_{0}pos=pos$ radians. A full circle is $2\pi \approx 6.283$, so the pair completes one loop when $pos$ crosses $6.283$ and then keeps going (the angle just keeps adding — sine/cosine wrap automatically, this is why values stay bounded no matter how large $pos$ gets).

• $pos=0$: angle $0$ → $(\sin ,\cos )=(0,+1)$, on the positive-$\cos$ axis.
• $pos=2$: angle $2$ rad (between $\pi /2\approx 1.571$ and $\pi \approx 3.14$) → Quadrant II: $\sin >0$ ($0.909$), $\cos <0$ ($-0.416$).
• $pos=4$: angle $4$ rad (between $\pi$ and $3\pi /2\approx 4.712$) → Quadrant III: $\sin <0$ ($-0.757$), $\cos <0$ ($-0.654$).
• $pos=6$: angle $6$ rad (between $3\pi /2$ and $2\pi \approx 6.283$) → Quadrant IV: $\sin <0$ ($-0.279$), $\cos >0$ ($+0.960$).
• $pos=7$: angle $7$ rad $>2\pi$; subtract one loop $7-2\pi \approx 0.717$ rad → back to Quadrant I: $\sin >0$ ($0.657$), $\cos >0$ ($+0.754$).

The point has walked all the way around and started a new lap. Key takeaway: the fast dimension revisits every quadrant quickly, so its sign alone never pins down absolute position — you need the slow dimensions (which are still deep in Quadrant I at these small angles) to say which lap you're on.

$PE(1)=[\sin 1,\cos 1,\sin 0.01,\cos 0.01]=[0.8415,0.5403,0.0100,0.99995]$. Add to embedding: $[-0.3+0.8415,0.4+0.5403,0.2+0.0100,-0.6+0.99995]=[0.5415,0.9403,0.2100,0.39995]$ At $pos=100$: the high-frequency dims $0,1$ (${\omega }_0=1$, angle $100$ rad $\approx 15.9$ full loops) have wrapped around many times — they look "randomly" placed on the fast circle, while the low-frequency dims $2,3$ (${\omega }_1=0.01$, angle $1$ rad) move only to $\sin 1=0.8415,\cos 1=0.5403$ — a modest, orderly change still in Quadrant I. Fast dims resolve exact neighbourhood; slow dims track that we're "far into the document."

• RoPE: instead of adding a sinusoid then hoping a linear layer learns the rotation ${\mathbf{T}}_k$, RoPE directly rotates the query and key vectors by ${\omega }_i\cdot pos$, baking the relative-position rotation into attention by construction — see Rotary Position Embeddings (RoPE).
• Relative encodings: L5.1's "dot product depends only on distance" is precisely the property Relative Positional Encodings make explicit, encoding $po{s}_1-po{s}_2$ rather than absolute positions.

Even index means which function? ::: Sine. ${\omega }_i$ formula? ::: $1/10000^{2i/d_{model}}$. What geometric object is ${\mathbf{T}}_k$? ::: A rotation matrix (by angle ${\omega }_{i}k$). $PE(0)$ pattern? ::: Alternating $0,1,0,1,\dots$. Pair dot product equals? ::: $\cos ({\omega }_i\cdot \text{distance})$. Why do sin/cos values stay bounded as $pos$ grows huge? ::: The angle just wraps around the circle every $2\pi$.