Foundations — Vector databases and embeddings

The parent note Vector databases and embeddings throws a lot of notation at you in one breath: ${\mathbb{R}}^d$, $\vec{a}$, $\Vert \vec{a}\Vert$, $\sum$, $\cos \theta$, $O(N\cdot d)$. If any of those made you pause, this page is for you. We build each one from absolutely nothing, anchor it to a picture, and only then let the next symbol lean on it.

1. A number line, then a plane — what "space" means

Now glue two number lines together at right angles — one horizontal, one vertical. Any point now needs two numbers to pin it down: how far right, how far up. That pair, like $(3,4)$, is a location on a flat sheet (a plane).

WHAT we just did: we upgraded from "one number = one position on a line" to "two numbers = one position on a plane." WHY the topic needs it: an embedding is a long list of numbers, not just two. But the leap from 1 number to 2 is the same leap as from 2 to 768 — you just keep adding axes. If you understand the plane, you understand the idea; the rest is scale.

2. The arrow: what a vector is

Look at the figure above: the arrow labelled $\vec{a}$ starts at the origin and its tip lands at $(3,4)$. Two facts about an arrow matter to us:

• Direction — which way it points.
• Length — how far it reaches.

3. Coordinates and the subscript $a_i$

WHY we need this: we cannot write out 768 separate letters. The subscript lets us say "the $i$-th number of $\vec{a}$" once and mean all of them. It is bookkeeping, nothing deeper.

4. Dimension $d$ and the space ${\mathbb{R}}^d$

5. The summation symbol $\sum$

Worked unrolling, so it is never mysterious again. For $d=3$: ${\sum }_{i=1}^3{a}_i=a_1+a_2+a_3$ That is the entire meaning — a loop that adds. WHY the topic needs it: dot products and norms both add up many small pieces, and writing $a_1{b}_1+a_2{b}_2+\cdots +a_{768}{b}_{768}$ by hand is absurd. $\sum$ says it in three symbols.

6. Length of an arrow: the norm $\Vert \vec{a}\Vert$

Pythagoras says: (long side)${}^2$ = (across)${}^2$ + (up)${}^2$. So the length is $\Vert \vec{a}\Vert =\sqrt{a_1^2+a_2^2}=\sqrt{3^2+4^2}=\sqrt{9+16}=\sqrt{25}=5.$

WHY the topic needs it: two documents about dogs — a tweet and a whole essay — can point the same direction but have very different lengths. To compare meaning we must be able to measure and then cancel out length. The norm is the measuring stick. More in Vector Norms.

7. The angle $\theta$ between two arrows

WHY the topic needs it: the angle is the similarity signal. Look at the figure: the two mint arrows point almost the same way (tiny $\theta$, very similar), while the coral arrow points off at a wide $\theta$ (unrelated). Turning "angle" into "a number between $-1$ and $1$" is the job of cosine, next.

8. Cosine: turning an angle into a similarity number

You don't need trigonometry to use it, because — as the parent shows — cosine can be computed straight from the coordinates via the dot product, without ever finding the angle first. That is the magic of the next symbol.

9. The dot product $\vec{a}\cdot \vec{b}$

Tiny example: $(1,2,2)\cdot (2,0,1)=1\cdot 2+2\cdot 0+2\cdot 1=2+0+2=4$.

The deep fact (proven in Dot Product) is that this arithmetic secretly equals length-times-length-times-cosine: $\vec{a}\cdot \vec{b}=\Vert \vec{a}\Vert \Vert \vec{b}\Vert \cos \theta .$

WHY this matters so much: rearrange it and you get the whole parent page's centerpiece, $\cos \theta =\frac{\vec{a}\cdot \vec{b}}{\Vert \vec{a}\Vert \Vert \vec{b}\Vert }=\underset{\text{meaning-closeness}}{\underbrace{\text{[[Cosine Similarity|cosine similarity]]}}}.$ We divide by both lengths precisely to cancel magnitude (from §6) and leave only direction (from §7–8). The dot product is the cheap, coordinate-only route to the angle — no trigonometry needed. Databases love it because it is just "multiply and add," which hardware does at blazing speed.

10. Reading $O(N\cdot d)$ — counting the work

Here $N$ = how many stored vectors, $d$ = length of each vector. To compare a query against all $N$ stored vectors, each comparison touches $d$ numbers → total $\approx N\cdot d$ operations. For $N=10^9$ and $d=768$ that is nearly a trillion operations per query — too slow. That single realisation is why the parent introduces approximate search (HNSW and ANN Indexes); $O(\log N)$ just means the work grows far more gently as $N$ explodes.

How these foundations feed the topic

Read it top-down: the arrow idea splits into how we address its numbers (index → sum), how big the space is (dimension → cost), and how we measure it (norm, dot, angle → cosine → similarity). All roads meet at the parent topic.

Equipment checklist

Cover the right side and answer out loud; reveal to check.

Connections

• Parent topic: Vector databases and embeddings — everything here feeds it.
• Dot Product · Vector Norms · Cosine Similarity — the three measuring tools built above.
• Curse of Dimensionality — where high $d$ stops behaving like the 2D picture.
• HNSW and ANN Indexes — why $O(N\cdot d)$ forces approximate search.
• Retrieval-Augmented Generation · Prompt Engineering · Chunking Strategies · Contrastive Learning — the pipeline these foundations power.