4.1.6 · D5Transformer Architecture
Question bank — Positional encodings (sinusoidal)
This page is a set of thinking questions about sinusoidal positional encodings. Nothing here needs a calculator — every trap is about understanding why the formula is shaped the way it is. Read the prompt, answer out loud, then reveal.
Before you start, three words we lean on:
- permutation invariant — shuffle the inputs and the output is unchanged. Plain picture: a bag of tiles with no slots; rearranging tiles in the bag looks identical.
- frequency — how fast a sine wave wiggles as position grows. High = tight ripples; low = slow swells.
- wavelength — how many positions you march before the wave repeats one full cycle.
True or false — justify
True or false: Without positional encoding, self-attention could still tell "dog bites man" from "man bites dog."
False. Attention is permutation invariant, so swapping token positions leaves every attention score identical; the two sentences become indistinguishable.
True or false: Positional encodings are concatenated onto token embeddings.
False. They are added element-wise, so the vector keeps its dimension ; addition lets each dimension blend meaning and position rather than reserving separate slots.
True or false: A plain counter would encode position just as safely as sines.
False. A counter grows without bound, so far positions would dominate the summed vector and swamp the semantic embedding; sines stay inside .
True or false: Every embedding dimension oscillates at the same frequency.
False. Each dimension uses , a decreasing frequency, so low dimensions wiggle fast (local order) and high dimensions swell slowly (global position).
True or false: The sine and cosine within one pair share the same frequency.
True. They are and with the same ; together they trace a point on one circle, which is exactly what makes relative shifts a clean rotation.
True or false: Sinusoidal PE must be re-trained for sequences longer than those seen in training.
False. The formula is a fixed function of , so it extrapolates to any position; this is a key edge over learned embeddings, which are capped at a maximum length.
True or false: Two different positions can accidentally receive the exact same full PE vector.
Practically false within any real sequence. Because the slowest dimension has wavelength near , no two positions inside that range collide across the whole vector, even though a single dimension repeats.
True or false: Adding PE destroys the token's meaning because you're changing its embedding numbers.
False. The change is a small, structured, position-only offset; the model learns to read both the semantic direction and the positional signal from the summed vector, so meaning survives.
True or false: The matrix that maps to depends on the absolute position .
False. It is a rotation by angle that depends only on the offset (and ), never on where you started — that position-independence is the whole point.
Spot the error
"PE tells the model which word is at each position." — what's wrong?
PE encodes position only; the identity of the word comes from the token embedding. PE at position 3 is the same whether the word there is "cat" or "quantum."
"Even dimensions use cosine, odd dimensions use sine." — what's wrong?
It's reversed: even indices () use , odd indices () use . The convention matters so the pair lands correctly for the rotation identity.
"We use two frequencies because one for sine and one for cosine." — what's wrong?
The pair uses one frequency for both and . The "two functions" give a 2D point on a circle; using different frequencies would break the neat angle-addition shift.
"The base 10000 was chosen because sequences are never longer than 10000 tokens." — what's wrong?
10000 sets the spread of wavelengths (shortest to longest positions), not a hard length cap. It balances fine local resolution against long-range reach; the encoding still works beyond 10000.
"Since sine repeats every , positions past get confused." — what's wrong?
A single high-frequency dimension does repeat, but the combination of many decreasing frequencies is unique over the useful range. No single wave carries the whole burden.
"Because PE uses trig, you must feed the model radians in degrees first." — what's wrong?
The argument is already a pure number in radians by construction; there is no degree conversion — degrees never enter the formula.
"Concatenating PE would be strictly better since it keeps position and meaning separate." — what's wrong?
Concatenation would double the dimension and force every downstream weight matrix to grow; addition keeps size fixed and empirically lets attention recover position via learned projections just fine.
Why questions
Why do we need both sine and cosine instead of sine alone?
One value is ambiguous (many angles share a sine); pairing it with pins the angle uniquely on the circle and makes a shift a linear rotation via the angle-addition identity.
Why do frequencies decrease across dimensions rather than increase or stay flat?
So the encoding covers many "zoom levels": fast dimensions separate neighbouring tokens, slow dimensions track coarse global position, giving a smooth geometric ladder of resolutions.
Why does the shift turning into a rotation matter for the model?
A linear layer can learn that fixed rotation, so it can compute " steps away" the same way at every position — this is how absolute encodings quietly support relative reasoning (see Relative Positional Encodings).
Why are sines chosen over an unbounded but unique function like ?
Boundedness. Trig stays in regardless of length, so long sequences never blow up the summed embedding magnitude, and gradients stay well-scaled.
Why is this connected to the Fourier Transform idea?
Because PE decomposes position into a bank of sinusoids at many frequencies — literally a Fourier-style basis where each dimension is one frequency component of the position signal.
Why can Rotary Position Embeddings (RoPE) be seen as a cousin of this scheme?
RoPE applies the same per-frequency rotation directly to query/key vectors inside attention, reusing the rotation-by- insight that Worked Example 2 exposed, but multiplicatively rather than additively.
Edge cases
At , what does the full PE vector look like and why?
Every and every , so alternates . Position zero is a clean, fixed anchor — not a special case, just the natural start of every wave.
For the very highest dimension (largest ), why does its value barely move between adjacent positions?
Its frequency is tiny, so changes almost nothing per step; that dimension is a slow global clock, meant to differ over thousands of positions, not one.
What happens conceptually if you set the base to 1 instead of 10000?
All exponents collapse (), so every dimension shares the single fastest frequency; you lose the multi-scale ladder and distant positions become nearly indistinguishable at coarse scale.
If two positions are exactly one wavelength apart in some dimension, are they identical?
In that one dimension, yes — the wave returns to the same value. But other dimensions with different wavelengths still differ, so the full vectors remain distinct.
For a sequence of length 1 (single token), does positional encoding do anything useful?
It adds , but with only one token there is no order to disambiguate; the encoding is harmless and the model simply relies on the semantic embedding.
Recall Fast self-test
The one-frequency limitation ::: A single sine repeats every wavelength, so distant positions look alike — solved by using many decreasing frequencies. The rotation insight ::: is a rotation of by , independent of , enabling learnable relative offsets. Add vs concat ::: PE is added, keeping dimension fixed while blending meaning and position.