Positional encodings (sinusoidal)

Transformers process all tokens in parallel, losing the sequential order information that RNNs naturally preserve. Positional encoding injects position information into token embeddings so the model knows "which word came first."

$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)$

$PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$

Even dimensions use sine, odd dimensions use cosine. These are added (not concatenated) to the token embeddings.

Derivation: Why This Formula?

Step 1: Need something that changes with position We want $PE(pos)$ to be unique for each position. A simple counter $[1, 2, 3, ...]$ would work but scales unboundedly—position 1000 would dominate the embedding values.

Step 2: Bounded functions—sine and cosine Trig functions oscillate in $[-1, 1]$, keeping values bounded. For a single frequency $\omega$:

$PE(pos) = \sin(\omega \cdot pos)$

Why this step? Sine gives us periodicity. Different positions get different values, but the function repeats. One frequency isn't enough because it repeats (period $2\pi/\omega$), making distant positions look similar.

Step 3: Multiple frequencies for different dimensions Use a different frequency for each embedding dimension. Dimension $i$ gets frequency:

$\omega_i = \frac{1}{10000^{2i/d_{model}}}$

Why decreasing frequencies?

• Low dimensions ($i$ small): high frequency $\rightarrow$ changes rapidly with position $\rightarrow$ captures fine-grained local order
• High dimensions ($i$ large): low frequency $\rightarrow$ changes slowly $\rightarrow$ captures coarse global position

The factor $10000$ sets the wavelength range. The longest wavelength is $2\pi \cdot 10000 \approx 62832$ positions.

Step 4: Alternating sine and cosine Even dimensions use sine, odd use cosine of the same frequency:

$PE_{(pos, 2i)} = \sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)$ $PE_{(pos, 2i+1)} = \cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)$

Why both? The pair $(\sin(\omega pos), \cos(\omega pos))$ forms a 2D representation of position on a circle of frequency $\omega$. This allows the model to learn linear transformations to compute relative positions using trig identities:

$\sin(\omega(pos + k)) = \sin(\omega pos)\cos(\omega k) + \cos(\omega pos)\sin(\omega k)$

A linear layer can learn weights $\cos(\omega k)$ and $\sin(\omega k)$ to shift any position by $k$ steps.

Relative Position: For any fixed offset $k$, $PE(pos+k)$ is a linear function of $PE(pos)$:

$PE(pos+k) = \mathbf{T}_k \cdot PE(pos)$

where $\mathbf{T}_k$ is a linear transformation. This makes relative distances learnable.

Bounded Values: All components stay in $[-1, 1]$.

Extrapolation: Works for positions longer than training sequences (unlike learned positional embeddings capped at max length).

Worked Example 1: Computing PE for a 4-Dimensional Model

Let $d_{model} = 4$, compute $PE(pos=3)$.

Dimensions: $i \in \{0, 1\}$ (since $2i$ and $2i+1$ cover indices 0,1,2,3).

Dimension 0 (i=0, even → sine): $PE_{(3,0)} = \sin\left(\frac{3}{10000^{0/4}}\right) = \sin(3) = 0.1411$

Dimension 1 (i=0, odd → cosine): $PE_{(3,1)} = \cos\left(\frac{3}{10000^{0/4}}\right) = \cos(3) = -0.9900$

Dimension 2 (i=1, even → sine): $\omega = \frac{1}{10000^{2/4}} = \frac{1}{100} = 0.01$ $PE_{(3,2)} = \sin(3 \cdot 0.01) = \sin(0.03) = 0.0300$

Dimension 3 (i=1, odd → cosine): $PE_{(3,3)} = \cos(3 \cdot 0.01) = \cos(0.03) = 0.9996$

Why these steps? Lower dimensions (0,1) oscillate rapidly (high $\omega=1$), capturing exact position. Higher dimensions (2,3) oscillate slowly (low $\omega=0.01$), capturing coarse position within sequence.

Final: $PE(3) = [0.1411, -0.9900, 0.0300, 0.9996]$

Worked Example 2: Demonstrating Relative Position Linearity

Show that $PE(pos+1)$ relates linearly to $PE(pos)$ for dimension pair $(2i, 2i+1)$.

For dimension $i$, let $\omega_i = 1/10000^{2i/d_{model}}$. We have:

$PE_{(pos,2i)} = \sin(\omega_i \cdot pos), \quad PE_{(pos,2i+1)} = \cos(\omega_i \cdot pos)$

For position $pos+1$:

$PE_{(pos+1,2i)} = \sin(\omega_i(pos+1)) = \sin(\omega_i pos + \omega_i)$

Using angle addition: $= \sin(\omega_i pos)\cos(\omega_i) + \cos(\omega_i pos)\sin(\omega_i)$ $= PE_{(pos,2i)} \cdot \cos(\omega_i) + PE_{(pos,2i+1)} \cdot \sin(\omega_i)$

Why this step? The trig identity converts the new position into a weighted sum of the old position's encoding. The weights $\cos(\omega_i)$ and $\sin(\omega_i)$ depend only on the offset (1 step), not the absolute position.

Similarly for the cosine dimension: $PE_{(pos+1,2i+1)} = \cos(\omega_i pos)\cos(\omega_i) - \sin(\omega_i pos)\sin(\omega_i)$ $= PE_{(pos,2i+1)} \cdot \cos(\omega_i) - PE_{(pos,2i)} \cdot \sin(\omega_i)$

In matrix form: $\begin{bmatrix} PE_{(pos+1,2i)} \\ PE_{(pos+1,2i+1)} \end{bmatrix} = \begin{bmatrix} \cos(\omega_i) & \sin(\omega_i) \\ -\sin(\omega_i) & \cos(\omega_i) \end{bmatrix} \begin{bmatrix} PE_{(pos,2i)} \\ PE_{(pos,2i+1)} \end{bmatrix}$

This is a rotation matrix! A +1 offset rotates the 2D representation by angle $\omega_i$. The model can learn this transformation to attend to relative positions.

Worked Example 3: Why 10000 as the Base?

The wavelength of dimension $i$ is:

$\lambda_i = 2\pi \cdot 10000^{2i/d_{model}}$

For $d_{model}=512$:

• Dimension 0 ($i=0$): $\lambda_0 = 2\pi \cdot 10000^0 = 2\pi \approx 6.28$ (high frequency)
• Dimension 255 ($i=255$): $\lambda_{255} = 2\pi \cdot 10000^{510/512} \approx 2\pi \cdot 9930 \approx 62393$ (low frequency)

(Note: $10000^{510/512}$ is slightly less than $10000$, so the maximal wavelength is just under $2\pi \cdot 10000 \approx 62832$.)

Why this range?

• Short wavelengths ($\approx 6.28$) distinguish adjacent positions (every position looks different)
• Long wavelengths ($\approx 62393$) encode global position in long documents (sequences up to ~10k tokens)

The geometric progression $10000^{2i/d_{model}}$ ensures smooth coverage across scales—each dimension handles a specific "zoom level" of position.

Why this step? Too small a base (e.g., 100) → longest wavelength only ~628 positions → fails on long sequences. Too large → spectrum stretches too much, wasting resolution. The value $10000$ balances fine local resolution with long-range coverage.

Step 1: Embed tokens into vectors (from embedding table):

• "The" → $[0.2, -0.5, 0.8, 0.1]$
• "cat" → $[-0.3, 0.4, 0.2, -0.6]$
• "sat" → $[0.7, 0.1, -0.4, 0.3]$

Step 2: Compute positional encodings for positions 0,1,2 (using $d_{model}=4$):

For $pos=0$:

• $PE_{(0,0)} = \sin(0) = 0$
• $PE_{(0,1)} = \cos(0) = 1$
• $PE_{(0,2)} = \sin(0) = 0$
• $PE_{(0,3)} = \cos(0) = 1$
• $PE(0) = [0, 1, 0, 1]$

For $pos=1$:

• $PE_{(1,0)} = \sin(1) = 0.8415$
• $PE_{(1,1)} = \cos(1) = 0.5403$
• $PE_{(1,2)} = \sin(0.01) = 0.0100$
• $PE_{(1,3)} = \cos(0.01) = 0.99995$
• $PE(1) = [0.8415, 0.5403, 0.0100, 0.99995]$

For $pos=2$:

• $PE(2) = [0.9093, -0.4161, 0.0200, 0.9998]$

Step 3: Add positional encodings to embeddings:

• "The" + $PE(0)$ = $[0.2, 0.5, 0.8, 1.1]$
• "cat" + $PE(1)$ = $[0.5415, 0.9403, 0.2100, 0.39995]$
• "sat" + $PE(2)$ = $[1.6093, -0.3161, -0.3800, 1.2998]$

These position-aware embeddings feed into the first transformer layer.

Why this step? Addition (not concatenation) preserves dimensionality and lets the model blend semantic content (embeddings) with positional information (encodings).

Why it feels right: Since we compute PE per position, it seems like it's encoding the token itself.

The fix: Positional encoding ONLY encodes where the token is, not what it is. The token embedding encodes meaning ("cat" vs "dog"). They're added together:

$\text{Input} = \text{Embedding}(\text{token}) + PE(\text{position})$

Without the embedding, the model has no idea what words mean. Without PE, it has no idea what order they're in.

Why it feels right: Concatenation keeps information distinct: $[embedding; PE]$ has both components intact.

The fix: Concatenation doubles dimensionality ($d_{model}$ → $2 \cdot d_{model}$), increasing compute cost. Addition works because:

1. Embeddings and PE live in the same $d_{model}$ space
2. Self-attention learns to disentangle them via learned projections $W_Q, W_K, W_V$
3. PE values are small (bounded in [-1,1]) compared to embeddings (initialized with larger variance), so they "nudge" rather than overwhelm

Empirically, addition works as well as concatenation with half the parameters.

Why it feels right: The index $i$ runs from 0 to $d_{model}-1$, same as positions 0 to $d_{model}-1$.

The fix: Each dimension $i$ encodes a frequency component present at ALL positions. Dimension 0 oscillates quickly at every position (high frequency). Dimension 511 oscillates slowly at every position (low frequency). Think of it like a Fourier transform: low frequencies capture the big picture, high frequencies capture details. Every position uses all dimensions.

A Transformer is like a super-smart reader that looks at all panels once. But if we just give it the panels without numbers, it can't tell the story correctly. So we write a secret code on each panel that says "this is panel 3" without actually writing "3".

The code is a pattern of waves—like a fingerprint made of wiggly lines. Panel 1 gets one fingerprint, panel 2 gets another, and so on. Some parts of the fingerprint wiggle fast (to tell nearby panels apart), and some wiggle slow (to tell if you're near the start or end of the comic). We add this fingerprint to each panel, so the Transformer knows the order while still seeing all the pictures at once.

• Fixed (not learned)
• Addition to embeddings
• Sine on even, cosine on odd
• Ten-thousand base (10000)

Visual: Imagine a spiral notebook where each coil (dimension) has a different tightness—tight coils for fine position, loose coils for coarse position.

Connections

• Token Embeddings: PE is added to these to create position-aware inputs
• Self-Attention Mechanism: PE enables attention to distinguish "attend to previous word" from "attend to next word"
• Multi-Head Attention: Different heads can learn to focus on different relative position offsets
• Learned Positional Embeddings: Alternative to sinusoidal; learned lookup table (limited to training sequence length)
• Relative Positional Encodings: Modern alternative (T5, Transformer-XL) that directly encodes relative distances in attention
• Fourier Transform: Sinusoidal PE decomposes position into frequency components, analogous to frequency analysis
• Rotary Position Embeddings (RoPE): Modern variant that applies rotation in attention space rather than adding to embeddings

#flashcards/ai-ml

Concept Map

Hinglish (regional understanding)

Hinglish (regional understanding)