Foundations — Positional encodings (sinusoidal)

Before you can read the parent note comfortably, you need a small toolbox of pictures. This page builds every symbol it uses, in order, from nothing. Each block gives you: plain meaning → the picture → why the topic needs it.

---

1. A position is just a counting number

Picture: a row of boxes. "The" goes in box $0$, "cat" in box $1$, "sat" in box $2$.

Why the topic needs it: the whole problem is that self-attention (see Self-Attention Mechanism) does not know box order — shuffle the boxes and it gives the same answer. So $pos$ is the thing we must somehow feed in.

---

2. A vector is a short list of numbers

Picture: an arrow, or simpler — a vertical stack of little bars, one bar per number. A 4-long vector is 4 stacked bars.

Why the topic needs it: every word becomes such a vector (a token embedding), and the positional encoding we build is also a vector of the same length $d_{model}$. We must match lengths so we can add them.

Recall Why must PE have length

$d_{model}$? Because we add it to the token embedding, and you can only add two lists of numbers slot-by-slot if they are the same length ::: PE and the embedding are both $d_{model}$ long so the sum is defined.

---

3. Adding two vectors

Picture: lay the two stacks of bars side by side; the result is a new stack where each bar is the two bars piled on top of each other.

Why the topic needs it: the parent note says PE is added (not glued on) to the embedding. Addition keeps the length at $d_{model}$, so the next layer sees a vector of the size it expects.

---

4. Sine and cosine — the two waves

This is the heart of everything, so we go slowly.

Why sine and cosine and not something else? The topic needs a function that:

1. never blows up (a raw counter $1,2,3,\dots$ would make $pos=1000$ swamp everything);
2. gives smoothly changing, distinguishable values.

A dot on a circle never leaves the circle, so its height and sideways position always stay between $-1$ and $+1$. That boundedness is exactly property (3) the parent demands.

Recall What is the range of

$\sin$ and $\cos$? Both always lie in ::: $[-1,1]$.

---

5. Radians (the angle unit)

Picture: wrap a string of length $1$ around the rim of the unit circle — the angle it spans is exactly $1$ radian.

Why the topic needs it: every $\sin (\cdot )$ and $\cos (\cdot )$ in the formula takes its input in radians. In Worked Example 1, $\sin (3)$ means "$3$ radians," a bit less than half a turn — that's why $\sin (3)=0.1411$ is small and $\cos (3)=-0.9900$ is near $-1$.

---

6. Period, frequency $\omega$, and wavelength $\lambda$

Picture: two waves drawn over positions $0\dots 60$. A fast wave (big $\omega$) wiggles many times — good for telling neighbours apart. A slow wave (small $\omega$) barely moves — good for tracking coarse, long-range position.

Why the topic needs it: the whole trick is giving each embedding dimension a different $\omega$, from fast to slow, so the vector reads position at many "zoom levels" at once.

Recall If frequency

$\omega$ is large, is the wavelength long or short? Short — a fast wave repeats quickly ::: $\lambda =2\pi /\omega$, so big $\omega$ means small $\lambda$.

---

7. Powers, exponents, and $10000^{2i/d_{model}}$

Picture: a ruler that is stretched — equal steps in the exponent become ever-bigger jumps in the value. This is a geometric (multiplicative) spacing.

Why the topic needs it: the frequencies are set by ${\omega }_i=\frac{1}{10000^{2i/d_{model}}}.$ As the dimension index $i$ climbs, the exponent $2i/d_{model}$ climbs from $0$ up toward $1$, so the denominator climbs from $1$ up to $\approx 10000$, so ${\omega }_i$ shrinks from $1$ down to $\approx \frac{1}{10000}$. That geometric sweep is how one vector spans "fast neighbour-detector" to "slow document-scale" frequencies. Check: at $i=0$, $10000^0=1$ so ${\omega }_0=1$; at the halfway exponent, $10000^{1/2}=100$ so $\omega =0.01$ — exactly the numbers in the parent's Worked Example 1.

---

8. Index $i$ and the even/odd split ($2i$ and $2i+1$)

Picture: the stack of bars from Figure 1, coloured in pairs — each colour pair is one $(\sin ,\cos )$ couple riding the same wave speed.

Why the topic needs it: keeping $\sin$ and $\cos$ of the same $\omega$ side by side is what stores the full 2-D circle address (Section 4). That pairing is what makes shifting position a clean rotation — the key relative-position property, and the seed idea behind Rotary Position Embeddings (RoPE).

---

9. The rotation matrix (a preview of the payoff)

Picture: the dot on the circle, spun forward by a fixed angle.

Why the topic needs it: stepping from position $pos$ to $pos+k$ multiplies each $(\sin ,\cos )$ pair by exactly this matrix with $\varphi ={\omega }_{i}k$. So "move $k$ steps" = "rotate by a fixed angle" — a linear rule the model can learn, giving consistent relative distances. (The parent's Worked Example 2 derives this.)

---

10. $P{E}_{(pos,j)}$ — reading the subscript

Why the topic needs it: it lets the formula name one exact number. Even $j$ picks the sine rule; odd $j$ picks the cosine rule — the split from Section 8.

---

How these pieces feed the topic

The parent topic sits at node F; everything on this page is one of the arrows leading in. Note how Token Embeddings enters only at the add step — PE never replaces the word's meaning, it rides alongside it.

---

Equipment checklist

Cover the right side and check you can answer each before reading the parent note.

• What does $pos$ count, and where does it start? ::: Which slot a word occupies; it starts at $0$.
• What is $d_{model}$? ::: The length (number of slots) of every vector the model uses.
• How do you add two vectors? ::: Slot by slot; both must be the same length.
• What are $\sin (\theta )$ and $\cos (\theta )$ geometrically? ::: The height and sideways position of a dot on the unit circle after turning angle $\theta$.
• What range do $\sin$ and $\cos$ always stay in? ::: $[-1,1]$.
• What is a radian, and how many make a full turn? ::: An angle measured by arc length on the unit circle; $2\pi$ make a full turn.
• What links frequency $\omega$ and wavelength $\lambda$? ::: $\lambda =2\pi /\omega$ — big $\omega$ means short $\lambda$.
• What is $10000^0$ and $10000^{1/2}$? ::: $1$ and $100$.
• As dimension index $i$ grows, does ${\omega }_i$ grow or shrink? ::: It shrinks (waves get slower).
• Why keep $\sin$ and $\cos$ of the same frequency together? ::: They give the full 2-D circle address, so a position shift becomes a clean rotation.
• What does $P{E}_{(3,2)}$ mean? ::: Slot $2$ of the encoding vector for position $3$.