Positional encodings (sinusoidal): Visual walkthrough

The parent note handed you the finished formula. Here we grow it from nothing — one wave at a time — until the whole thing stands on its own. Every step is a picture first, symbols second.

Step 1 — A row of slots that need name-tags

WHAT. Picture a sentence laid out as boxes: slot $0$ , slot $1$ , slot $2$ , … We call the slot number == $p os$ == (short for position) — just an ordinary counting number, $0, 1, 2, \dots$

WHY. Self-attention is permutation invariant: shuffle the boxes and the math gives the identical answer. So order is invisible unless we paint something different into each box. The simplest first idea — write the raw counter $p os$ itself — is drawn in the figure.

PICTURE.

Figure — Positional encodings (sinusoidal)

Look at the tall red bar over slot $9$ : the raw counter grows without limit. In a real document $p os$ could be $10, 000$ . That single huge number would drown the token embedding it gets added to. So the raw counter fails property "must stay small."

Step 2 — Tame the number with a wave

WHAT. Replace the ever-growing counter with the height of a wave as we walk along the slots. A wave is just the function $sin$ : feed it an angle, it hands back a height that forever swings between $- 1$ and $+ 1$ . We call this first tag == $P E (p os)$ == — the Positional Encoding at position $p os$ .

WHY. $sin$ answers exactly the complaint from Step 1 — "keep the tag bounded." No matter how big $p os$ gets, $sin$ never leaves $[- 1, 1]$ . That's why a trig function and not, say, $lo g$ or a raw ramp: we want something that both moves with position and never blows up.

Write our first tag as $P E (p os) = sin (how fast it swings ω \cdot which slot p os) .$

Here == $ω$ == (Greek "omega") is the angular frequency — how many radians of angle we advance per step. Big $ω$ = fast wiggle; small $ω$ = slow drift.

PICTURE.

The wave stays inside the two dashed lines at $\pm 1$ forever — problem solved. But notice the two orange dots: slot $2$ and slot $14$ sit at the same height. One wave repeats, so it hands out duplicate tags. That's the next thing to fix.

Step 3 — Two waves at once: sine and cosine

WHAT. Instead of one height per position, store two: the sine height and the cosine height, at the same frequency $ω$ . $(vertical height sin (ω p os), horizontal height cos (ω p os))$

WHY. == $cos$ == is just $sin$ shifted a quarter-turn: where sine is flat at its peak, cosine is steepest, and vice-versa. Pairing them means at every position at least one of the two is changing — the pair never repeats within a full turn. Geometrically the pair $(sin θ, cos θ)$ is a point on a circle of radius 1, and walking forward one slot rotates that point. Distance becomes angle, which is the whole trick we're after.

PICTURE.

Each dot is one position living on the unit circle. Stepping $p os \to p os + 1$ swings the dot by a fixed angle $ω$ (the little violet arc). Every slot is now a unique direction on the circle — no two positions collide until we've circled all the way round.

Recall Why exactly two, not three or one?

One number collides (Step 2). Two numbers put you on a circle where forward = rotate — the minimum needed to make "distance" a clean, learnable rotation. ::: Two.

Step 4 — Many circles: give each dimension its own frequency

WHAT. One (sin, cos) pair is one 2-D slice. An embedding has == $d_{m o d e l}$ == slots (its dimension), and we require $d_{m o d e l}$ to be even so the slots split cleanly into pairs. There are $d_{m o d e l} /2$ pairs, indexed by == $i$ == running $i = 0, 1, 2, \dots, \frac{d _{m o d e l}}{2} - 1$ . Fill them with many pairs, each pair spinning at a different speed. Pair number $i$ uses $ω_{i} = \frac{1}{1000 0 ^{2 i / d_{m o d e l}}}, i \in {0, 1, \dots, \frac{d _{m o d e l}}{2} - 1} .$ Read it left to right: $i$ is which pair; as $i$ climbs, the exponent grows, the denominator grows, so $ω_{i}$ shrinks — later pairs spin slower. At the ends: $i = 0$ gives $ω_{0} = 1$ (fastest); the last $i = \frac{d _{m o d e l}}{2} - 1$ gives the slowest wave.

WHY. A single frequency is either too fast (great for neighbours, useless for far apart) or too slow (opposite problem). Using a geometric ladder of frequencies is like keeping clocks that tick at every scale at once — a fast second-hand for local order, a slow year-hand for global position.

Why the base $10000$ ? It is a design dial that sets how far apart the fastest and slowest clocks tick. The slowest wavelength is roughly $2 π \cdot (base)$ positions, so the base must be big enough that even the longest realistic document (thousands of tokens) still fits inside less than one full turn of the slowest wave — otherwise the coarse "global position" clock would itself wrap around and collide. Pick it too small (say $100$ ) and the longest wave repeats after only $\approx 628$ slots, useless for long text; too large and the ladder stretches so far it wastes resolution in the middle. The Transformer authors chose $10000$ as the value that balances fine local resolution against long-range coverage for sequences up to $\sim 10, 000$ tokens (Step 6 checks the edges numerically).

PICTURE.

Top strip: pair $0$ (fast) flips sign every few slots — perfect for telling neighbours apart. Bottom strip: a high- $i$ pair (slow) barely drifts across the whole sentence — it labels coarse position. Stack all such strips and you get the classic striped heat-map of positional encodings.

Step 5 — Even slot = sine, odd slot = cosine (the final formula)

WHAT. Lay the pairs into the $d_{m o d e l}$ slots: even index $2 i$ gets the sine of pair $i$ , the next odd index $2 i + 1$ gets the cosine of the same pair $i$ :

PE_{(pos,\,2i+1)}=\cos\!\big(pos\cdot\omega_i\big).$$ **WHY.** Keeping sine and cosine of a frequency *adjacent* means each consecutive slot-pair is one full circle from Step 3. That's what lets a later [[Multi-Head Attention|attention]] layer shift positions with a plain linear map — because on a circle, "move forward $k$" is just "rotate by $k\omega_i$": $$\begin{bmatrix}PE_{(pos+k,2i)}\\ PE_{(pos+k,2i+1)}\end{bmatrix} =\underbrace{\begin{bmatrix}\cos(k\omega_i)&\sin(k\omega_i)\\ -\sin(k\omega_i)&\cos(k\omega_i)\end{bmatrix}}_{\text{rotation, depends only on offset }k} \begin{bmatrix}PE_{(pos,2i)}\\ PE_{(pos,2i+1)}\end{bmatrix}.$$ Every entry of that matrix depends on $k$ *alone*, never on $pos$ — so **relative** distance is baked in everywhere in the sentence. (This exact rotation idea is what [[Rotary Position Embeddings (RoPE)|RoPE]] later pushes further, and it's why these codes cousin the [[Fourier Transform|Fourier]] way of writing signals as sums of sines.) **PICTURE.** ![[deepdives/dd-ai-ml-4.1.06-d2-s05.png]] The circle shows $pos$ (magenta) and $pos+k$ (violet) on the *same* radius-1 ring; the shaded wedge is the fixed rotation $k\omega_i$. Same wedge whether you start near $0$ or near $1000$ — that's the property RNNs give for free and we've now rebuilt from waves. > [!example] Plug in real numbers ($d_{model}=4,\ pos=3$) > Here $d_{model}=4$ is even, so pairs are $i\in\{0,1\}$. > Pair $i=0$: $\omega_0=1/10000^{0}=1$, so > $PE_{(3,0)}=\sin 3=0.1411$, $PE_{(3,1)}=\cos 3=-0.9900$. > Pair $i=1$: $\omega_1=1/10000^{1/2}=0.01$, so > $PE_{(3,2)}=\sin(0.03)=0.0300$, $PE_{(3,3)}=\cos(0.03)=0.9996$. > Tag: $PE(3)=[0.1411,\,-0.9900,\,0.0300,\,0.9996]$. --- ## Step 6 — The edge cases: $pos=0$, the slowest wave, and beyond training length **WHAT.** Check the corners where things could break. - **$pos=0$.** Every sine is $\sin 0=0$, every cosine $\cos 0=1$. So $PE(0)=[0,1,0,1,\dots]$ — a clean, well-defined "start" tag, not a special case that crashes. - **Slowest wave.** For $d_{model}=512$, the last pair is $i=255$ with exponent $2i/d_{model}=510/512$. Its wavelength is $\lambda=2\pi\cdot 10000^{510/512}\approx 2\pi\cdot 9646\approx 60620$ slots — it barely completes one turn across a 60k-token book. That's the *deliberate* longest ruler. - **$pos$ longer than any training sentence.** The formula is a plain function of $pos$; feed it $pos=50000$ and it still returns a bounded, sensible vector. This is the *extrapolation* win over [[Learned Positional Embeddings|learned]] tables, which simply have no row for unseen positions. **WHY.** A tag scheme must never hit a slot it can't label. Because sine/cosine are defined for *every* real input and stay in $[-1,1]$, there is no "off the end of the table" and no blow-up — every corner is covered. **PICTURE.** ![[deepdives/dd-ai-ml-4.1.06-d2-s06.png]] Left: $PE(0)$ as a flat picket of $0$s and $1$s. Middle: the slowest wave creeping across thousands of slots. Right: two curves — the wave sails smoothly *past* the dashed "training length" line, while the learned table (dots) just stops. --- ## The one-picture summary ![[deepdives/dd-ai-ml-4.1.06-d2-s07.png]] Read it top to bottom: a raw counter (unbounded, bad) → one wave (bounded but repeats) → a sin/cos circle (unique, rotatable) → a ladder of frequencies filling all $d_{model}$ slots → the final striped code added onto the token embedding. > [!recall]- Feynman retelling — say it like a story > We have boxes of words the model reads all at once, so it can't feel order. We want to paint a *number-tag* (a **positional encoding**, $PE$) in each box. Plain counting numbers get huge, so instead we paint the **height of a wiggling wave** — waves stay small forever. One wave repeats, giving twins, so we paint **two** heights (a sine and a cosine), which is the same as putting each position as a dot on a **circle**; now stepping one word forward just *turns the dot a little* (a turn is measured in $\pi$: $2\pi$ is once around). One circle can't tell neighbours from far-apart words at the same time, so we use **many circles spinning at different speeds** — fast ones for "who's next to whom," slow ones for "where in the whole book," with the base $10000$ chosen so even the slowest clock doesn't wrap around inside a long document. Even slots hold the sine, odd slots the matching cosine, so "jump $k$ words" is always the *same rotation* everywhere in the sentence — the model can learn that turn with a simple linear layer. And because sine and cosine are defined for every number, position $0$, the longest wave, and words longer than anything seen in training all just work. --- **Prerequisites & neighbours:** [[Token Embeddings]] · [[Self-Attention Mechanism]] · [[Multi-Head Attention]] · [[Learned Positional Embeddings]] · [[Relative Positional Encodings]] · [[Fourier Transform]] · [[Rotary Position Embeddings (RoPE)]] · [[4.1.06 Positional encodings (sinusoidal) (Hinglish)|Hinglish version]]