Worked examples — Positional encodings (sinusoidal)

This page is the "hit every case" companion to Positional encodings (sinusoidal). The parent note built the formula and did three clean examples. Here we deliberately hunt down the awkward cases: position zero, huge positions, the highest frequency dimension vs. the lowest, the rotation-by-offset trick, and an exam twist. Every number below is machine-checked at the bottom.

Before anything, the facts we will reuse constantly. For a token at position $pos$ and a dimension-pair index $i$, define the base $b=10000$ — the single constant that sets the whole frequency ladder — and the angular frequency

${\omega }_i=\frac{1}{b^{2i/d_{model}}}=\frac{1}{10000^{2i/d_{model}}}.$

The index $i$ runs over $i=0,1,2,\dots ,\frac{d_{model}}{2}-1$ — that is, there are exactly $\frac{d_{model}}{2}$ distinct frequencies, one per $(\sin ,\cos )$ pair, and together the $2i$ and $2i+1$ channels fill all $d_{model}$ dimensions. (For our tiny $d_{model}=4$ this means $i\in \{0,1\}$: two frequencies, four channels.)

Read ${\omega }_i$ as "how fast the angle spins as position advances by one." The denominators are powers of the base $b$; a bigger $i$ means a bigger power, hence a smaller ${\omega }_i$ (slower spin). Then the even/odd pair is

$P{E}_{(pos,2i)}=\sin ({\omega }_i\cdot pos),P{E}_{(pos,2i+1)}=\cos ({\omega }_i\cdot pos).$

---

The scenario matrix

Every question this topic can pose falls into one of these cells. The examples afterward are tagged with the cell(s) they cover.

Cell	Case class	What makes it tricky	Example(s)
A	Position zero (degenerate)	$\sin 0=0,\cos 0=1$ — a fixed pattern, not "no encoding"	Ex 1
B	Highest-frequency dim ($i=0$, $\omega =1$)	Angle in radians = position; wraps fast	Ex 2
C	Lowest-frequency dim (large $i$)	Angle barely moves; near-linear	Ex 3
D	Two positions look identical? (aliasing)	One dim can repeat; full vector cannot	Ex 4
E	Relative shift = rotation (the ${\mathbf{T}}_k$ claim)	Offset $k$ maps to a rotation angle; sign of $k$	Ex 5
F	Large position / extrapolation ($pos=1000$)	Values stay bounded; no explosion	Ex 6
G	Real-world word problem	Choosing base/wavelength for a task	Ex 7
H	Exam twist (dot product of two encodings)	Depends only on the offset (incl. negative)	Ex 8

Throughout we use a tiny model, $d_{model}=4$, so $i\in \{0,1\}$: ${\omega }_0=1/10000^{0/4}=1$ and ${\omega }_1=1/10000^{2/4}=1/100=0.01$.

---

Forecast: guess the four numbers before reading on. Many people say "all zeros." Are they right?

1. Even dims use sine of ${\omega }_i\cdot 0=0$. Why this step? Every frequency times zero is zero, so both sine arguments are $0$, and $\sin 0=0$. $P{E}_{(0,0)}=\sin 0=0$, $P{E}_{(0,2)}=\sin 0=0$.
2. Odd dims use cosine of $0$. Why this step? $\cos 0=1$, independent of frequency, because a zero angle points along the positive $x$-axis on the circle. $P{E}_{(0,1)}=\cos 0=1$, $P{E}_{(0,3)}=\cos 0=1$.
3. Assemble: $PE(0)=[0,1,0,1]$. Why this step? The full encoding vector is built by laying the channels out in index order $0,1,2,3$, so we interleave dim 0 (sine), dim 1 (cosine), dim 2 (sine), dim 3 (cosine) into one $d_{model}$-length vector — that ordered concatenation is $PE(0)$.

Verify: Not all zeros — the cosine channels are pinned to $1$. So "position 0" is a very specific, non-null fingerprint. On the circle it is the point $(\cos 0,\sin 0)=(1,0)$: the rightmost point, for every frequency simultaneously.

---

Forecast: the angle here equals the position measured in radians. Since one full loop is $2\pi \approx 6.28$, guess which two consecutive positions sit almost on top of each other.

1. Angle = position. Why this step? With ${\omega }_0=1$, the argument is exactly $pos$ radians. So position advances the circle-angle one radian per token.
2. Evaluate: $\sin (0)=0,\sin (1)=0.8415,\sin (2)=0.9093,\sin (3)=0.1411,\sin (4)=-0.7568,\sin (5)=-0.9589,\sin (6)=-0.2794,\sin (7)=\mathrm{0.6570.}$
3. Find the wrap. Why this step? A full period is $2\pi \approx 6.2832$. Positions $0$ and $6.2832$ would coincide; $pos=0$ vs $pos=6$ ($6$ rad) are close but not equal ($\sin 6=-0.2794\ne 0$).

Look at Figure s02 below: the smooth blue curve is $\sin (pos)$; the orange dots are the integer positions $0\dots 7$ with their values labelled. The red dashed line marks one full period at $2\pi \approx 6.28$ — notice how the dots have not returned to their starting height by $pos=6$, which is the point of this example.

Verify: The fastest channel completes its circle in about $6.28$ positions — it is the "seconds hand." Distinct positions get distinct sine values within one period, but this single channel alone would repeat after $\approx 6.28$ tokens. That is exactly why we need slower channels (Ex 3).

---

Forecast: the wavelength is $2\pi /0.01\approx 628$ positions. Over just 100 tokens we traverse a small arc, so the curve should look nearly straight. Guess $\sin (1)$ vs the naive "$1$."

1. Small-angle regime. Why this step? For $pos=50$, angle $=0.5$ rad; for $pos=100$, angle $=1.0$ rad. These are modest arcs on the circle, so the sine is close to (but below) the angle itself.
2. Evaluate: $\sin (0)=0,\sin (0.5)=0.4794,\sin (1.0)=0.8415$.
3. Compare to linear. Why this step? A linear counter would give $0,0.5,1.0$; the sine gives $0,0.4794,0.8415$ — close early, then bending. The slow channel encodes coarse global position without exploding.

Verify: Over $0\to 100$ this channel moved only $0\to 0.8415$ (never leaving $[-1,1]$), while the fast channel (Ex 2) looped $\sim 16$ times. Fast = fine local order, slow = coarse global position. Exactly the "zoom levels" the parent described.

---

Forecast: the fast channel (${\omega }_0=1$) has period exactly $2\pi$, so it repeats between these two. Does that mean the whole vector repeats? Guess yes/no.

1. Fast channel repeats. Why this step? $\sin (1\cdot 0)=\sin (1\cdot 2\pi )=0$ and $\cos (0)=\cos (2\pi )=1$. So dims $0,1$ are identical at these two positions.
2. Check the slow channel. Why this step? Its frequency is ${\omega }_1=0.01$, so at $pos=2\pi$ its angle is $0.01\cdot 6.2832=0.06283$ rad, not $0$. Then $\sin (0.06283)=0.06279\ne 0$.
3. Conclusion. Why this step? Because at least one channel differs, the full vectors differ. Aliasing on a single channel does not cause a full-vector collision — the multi-frequency design is what guarantees uniqueness over any realistic range.

Verify: $PE(0)$ dim 2 $=0$; $PE(2\pi )$ dim 2 $=0.06279$. Distinct. The channels only all realign after the least common period, astronomically far away — far beyond any sequence length.

---

Forecast: the parent claimed $PE(pos+k)={\mathbf{T}}_{k}PE(pos)$ where ${\mathbf{T}}_k$ rotates by ${\omega }_{i}k$. So predict: take the sub-vector at $pos=3$, rotate by $0.05$ rad, and land exactly on the sub-vector at $pos=8$.

Axis-ordering reminder: as flagged in the warning callout, for matrix algebra we stack the sub-vector as $\left[\begin{array}{c}\sin \theta \\ \cos \theta \end{array}\right]$ — sine on top.

1. Compute the source sub-vector at $pos=3$, angle $\theta =0.03$. Why this step? We need the "before" point in matrix ordering. $\sin (0.03)=0.029996$ (top), $\cos (0.03)=0.99955$ (bottom).
2. Compute the target directly at $pos=8$, angle $0.08$. Why this step? This is the "after" point we must reproduce: $\sin (0.08)=0.079915$ (top), $\cos (0.08)=0.996802$ (bottom).
3. Why this exact matrix form? Why this step? We are not guessing the matrix — it is forced by the sine and cosine addition identities. Writing $\theta +\varphi$ with $\varphi ={\omega }_{1}k$: $\sin (\theta +\varphi )=\sin \theta \cos \varphi +\cos \theta \sin \varphi ,\cos (\theta +\varphi )=\cos \theta \cos \varphi -\sin \theta \sin \varphi .$ Read the top identity: the new sine is $(\cos \varphi )\cdot \sin \theta +(\sin \varphi )\cdot \cos \theta$ — so the top row of the matrix must be $[\cos \varphi ,\sin \varphi ]$ acting on $[\sin \theta ,\cos \theta {]}^{\top }$. Read the bottom: the new cosine is $(-\sin \varphi )\cdot \sin \theta +(\cos \varphi )\cdot \cos \theta$ — so the bottom row is $[-\sin \varphi ,\cos \varphi ]$. That is precisely why the sign sits on the lower-left: $\left[\begin{array}{c}\sin (0.08)\\ \cos (0.08)\end{array}\right]=\left[\begin{array}{cc}\cos 0.05& \sin 0.05\\ -\sin 0.05& \cos 0.05\end{array}\right]\left[\begin{array}{c}\sin (0.03)\\ \cos (0.03)\end{array}\right],\varphi ={\omega }_{1}k=\mathrm{0.05.}$
4. Negative offset $k=-5$. Why this step? Every case must be covered, including going backward. Then $\varphi ={\omega }_{1}k=-0.05$. Since $\cos (-0.05)=\cos (0.05)$ and $\sin (-0.05)=-\sin (0.05)$, the matrix becomes $\left[\begin{array}{cc}\cos 0.05& -\sin 0.05\\ \sin 0.05& \cos 0.05\end{array}\right]$ — the inverse rotation (opposite spin direction). It sends $pos=8$ back to $pos=3$, confirming shifts are invertible rotations for any sign of $k$.

Look at Figure s03 below: the blue arrow is the sub-vector at $pos=3$, the orange arrow at $pos=8$, and the red arc is the $0.05$-rad rotation carrying blue onto orange. (For $k=-5$ the arc would sweep the other way.)

Verify: top row $=\cos (0.05)\cdot 0.029996+\sin (0.05)\cdot 0.99955=0.079915=\sin (0.08)$. ✓ The angle $0.05$ depends only on $k$ (and its sign), never on the absolute position $3$. That is why relative distances — forward and backward — are learnable by a single fixed linear layer.

---

Forecast: a raw counter would give a value of $1000$ here, swamping the embeddings. Sinusoids can't. Guess the range of each component.

1. Fast channel. Why this step? ${\omega }_0=1$, so angle $=1000$ rad. That is $1000/(2\pi )\approx 159.15$ full loops. $\sin (1000)=0.82688$, $\cos (1000)=0.56238$.
2. Slow channel. Why this step? ${\omega }_1=0.01$, angle $=10$ rad $\approx 1.59$ loops. $\sin (10)=-0.54402$, $\cos (10)=-0.83907$.
3. Assemble: $PE(1000)=[0.82688,0.56238,-0.54402,-0.83907]$. Why this step? Same interleaving rule as Ex 1 — dims are laid out in index order (sine, cosine, sine, cosine), so we place the fast pair in slots $0,1$ and the slow pair in slots $2,3$ to form the single position-1000 vector.

Verify: every entry lies in $[-1,1]$ despite $pos=1000$. No explosion — this is exactly the extrapolation property: the formula is defined and bounded for positions far beyond any training length, unlike a lookup table capped at max length (Learned Positional Embeddings).

---

Forecast: guess whether the slowest channel finishes less than one loop over $8192$ tokens.

1. Derive the longest wavelength. Why this step? A channel of frequency $\omega$ has period (wavelength) $\lambda =2\pi /\omega$, because $\sin (\omega pos)$ repeats when $\omega pos$ advances by $2\pi$. The longest wavelength comes from the smallest frequency ${\omega }_{\min }$, which is the largest-$i$ channel. With ${\omega }_i=1/b^{2i/d_{model}}$, the smallest frequency uses the largest exponent $2i=d_{model}-2$, so ${\omega }_{\min }=\frac{1}{b^{(d_{model}-2)/d_{model}}},{\lambda }_{\max }=\frac{2\pi }{{\omega }_{\min }}=2\pi b^{(d_{model}-2)/d_{model}}.$ This is where the expression comes from — it is just "period $=2\pi /{\omega }_{\min }$" with the base $b$ substituted.
2. Base $b=10000$, $d_{model}=512$: the exponent is $(512-2)/512=510/512$, so $10000^{510/512}=9655$, giving ${\lambda }_{\max }=2\pi \cdot 9655=60650$ positions. Why this step? Compare to $8192$: $60650\gg 8192$, so the slowest channel completes far less than one loop across the whole document — no wrap, global order preserved. Base $10000$ suffices.
3. Base $b=100$: $100^{510/512}=98.24$, so ${\lambda }_{\max }=2\pi \cdot 98.24=617.3$ positions. Why this step? $617<8192$, so the slowest channel wraps $\sim 13$ times — distant positions alias and global order is lost. Base $100$ fails for this task.

Verify: base $10000$ gives ${\lambda }_{\max }\approx 60650>8192$ ✓; base $100$ gives $\approx 617<8192$ ✗. Recommendation: keep the standard base $10000$ (or larger for ultra-long context) — it is precisely the parent's "too small a base fails on long sequences," now quantified. For context lengths well beyond $\sim 60$k tokens you would raise the base further or switch to Rotary Position Embeddings (RoPE).

---

Forecast: gap is $p-q=5$, so predict the dot product $=\cos (0.01\cdot 5)=\cos (0.05)$, regardless of the specific $8$ and $3$.

Axis-ordering note: the dot product is order-independent, so it does not matter whether we stack $(\sin ,\cos )$ or $(\cos ,\sin )$ — we sum sine·sine plus cosine·cosine either way. We keep the sine-first ordering for consistency with Ex 5.

1. Write both sub-vectors (sine top, cosine bottom). Why this step? We need the two circle points. At $p=8$: $(\sin 0.08,\cos 0.08)=(0.079915,0.996802)$. At $q=3$: $(\sin 0.03,\cos 0.03)=(0.029996,0.999550)$.
2. Dot product. Why this step? Multiply matching channels and add: $\sin 0.08\sin 0.03+\cos 0.08\cos 0.03=0.079915\cdot 0.029996+0.996802\cdot 0.999550=\mathrm{0.998750.}$
3. Compare to the identity. Why this step? The cosine difference identity says $\sin a\sin b+\cos a\cos b=\cos (a-b)$, and here $a-b=0.08-0.03=0.05$, so this should equal $\cos (0.05)=0.998750$.
4. Negative gap ($p<q$). Why this step? Swap to $p=3,q=8$: then $p-q=-5$ and the identity gives $\cos ({\omega }_1(-5))=\cos (-0.05)$. Since cosine is even, $\cos (-0.05)=\cos (0.05)=0.998750$ — the same value. So the dot product is symmetric: forward and backward gaps of equal size score identically, exactly what you want for distance-based attention.

Verify: $0.998750=\cos (0.05)=\cos (-0.05)$ ✓. Because $\cos ({\omega }_i(p-q))$ depends only on $|p-q|$, attention scores built from these encodings naturally reflect relative distance — the seed idea behind Relative Positional Encodings and Self-Attention Mechanism's position sensitivity.

---

Recall Quick self-test

$PE(0)$ for $d_{model}=4$ is ::: $[0,1,0,1]$ — cosine channels pinned to 1, not all zeros. Why doesn't a single-channel repeat cause a full-vector collision? ::: Other channels use different frequencies, so at least one differs until an astronomically large common period. In matrix form, which entry of the 2D sub-vector goes on top? ::: The sine ($\sin \theta$); cosine goes below — that ordering makes the rotation matrix work. A position offset of $k$ acts on each 2D sub-vector as a ::: rotation by angle ${\omega }_{i}k$; negative $k$ is the inverse (opposite-direction) rotation. Why does the dot product depend only on $|p-q|$? ::: It equals $\cos ({\omega }_i(p-q))$ and cosine is even, so sign of the gap doesn't matter. Base $100$ vs $10000$ for 8192-token docs? ::: Base 100 wraps (${\lambda }_{\max }\approx 617$); base 10000 does not ($\approx 60650$).

Related: Token Embeddings (what PE is added to), Multi-Head Attention, Fourier Transform (the multi-frequency view), and the Hinglish note.