Exercises — Vector databases and embeddings

What we do: replace $\vec{a}\to 2\vec{a}$ and see what survives.

• Dot product: $(2\vec{a})\cdot \vec{b}=2(\vec{a}\cdot \vec{b})$ → doubles.
• Norm: $\Vert 2\vec{a}\Vert =2\Vert \vec{a}\Vert$ → doubles.
• Cosine: $\frac{2(\vec{a}\cdot \vec{b})}{2\Vert \vec{a}\Vert \Vert \vec{b}\Vert }$ — the two $2$'s cancel → unchanged.

Answer: (c) cosine similarity. This is the whole point of dividing by the norms: it strips out length so only meaning-direction remains.

• embedding = (i) function text→vector.
• chunk = (ii) focused passage.
• k-NN = (iii) exact $k$ closest.
• ANN = (iv) approximate but fast. See HNSW and ANN Indexes.

Answer: $-1$. The angle is $180°$ and $\cos 180°=-1$. Semantically: opposite meaning-direction (as unrelated-and-then-some as this space allows). $0$ = unrelated (right angle), $1$ = same direction.

Dot $=2\cdot 1+1\cdot 2+2\cdot 2=2+2+4=8$. Why: the dot product is defined as "multiply matched components, then sum" — it measures how much the two arrows overlap direction-wise before we account for length. Norms $\Vert \vec{a}\Vert =\sqrt{4+1+4}=3$, $\Vert \vec{b}\Vert =\sqrt{1+4+4}=3$. Why: norm is root-of-sum-of-squares, which is just Pythagoras extended to $3$ coordinates — the true length of the arrow. Cosine $\frac{8}{3\cdot 3}=\frac{8}{9}\approx 0.889$. Why: dividing the overlap by both lengths cancels magnitude, leaving pure $\cos \theta$. Angle $\approx \arccos (0.889)\approx 27°$ — quite similar.

(a) $6\cdot 3+8\cdot 4=18+32=50$. Why: same dot-product definition — pair up components, multiply, add. (b) $\Vert \vec{a}\Vert =\sqrt{36+64}=10$, $\Vert \vec{b}\Vert =\sqrt{9+16}=5$ (root-of-sum-of-squares lengths), so Cosine $=\frac{50}{10\cdot 5}=1$. Why: dividing the "big-looking" dot ($50$) by both lengths reveals the arrows are the same direction scaled up → perfect $\cos \theta =1$. This is Vector Norms doing its job.

Why this formula: for unit vectors, expand $\Vert \vec{a}-\vec{b}{\Vert }^2=\Vert \vec{a}{\Vert }^2+\Vert \vec{b}{\Vert }^2-2\vec{a}\cdot \vec{b}=1+1-2\cos \theta =2-2\cos \theta$. Plug in: $2-2(0.6)=2-1.2=0.8$. Take-away: on unit vectors, Euclidean distance and cosine give the same ranking — smaller distance ↔ larger cosine.

$\Vert \vec{q}\Vert =\sqrt{2}$.

• $C_1$: dot $=1\cdot 2+1\cdot 2+0=4$; $\Vert C_1\Vert =\sqrt{8}=2\sqrt{2}$; sim $=\frac{4}{\sqrt{2}\cdot 2\sqrt{2}}=\frac{4}{4}=1$.
• $C_2$: dot $=0+1+0=1$; $\Vert C_2\Vert =\sqrt{2}$; sim $=\frac{1}{\sqrt{2}\cdot \sqrt{2}}=\frac{1}{2}=0.5$.
• $C_3$: dot $=1+0+0=1$; $\Vert C_3\Vert =1$; sim $=\frac{1}{\sqrt{2}\cdot 1}\approx 0.707$. Ranking: $C_1(1.0)>C_3(0.707)>C_2(0.5)$. Top-1 = $C_1$, because it points in exactly the same direction as $\vec{q}$ (just longer).

Not meaningful. $M_1$ and $M_2$ define different coordinate axes; "direction" in one space has no fixed relation to direction in the other. The cosine is essentially the angle between two arrows drawn on two different maps — you'd expect scores scattered near $0$ (random orientation in high $d$), giving noise. Fix: embed query and documents with the same model. See Contrastive Learning for why each model carves its own space.

(a) $N\cdot d=10^8\times 768=7.68\times 10^{10}$ ops. (b) ${\log }_2(10^8)\approx 26.6$, so $768\times 26.6\approx 2.04\times 10^4$ ops. (c) Speedup $\approx \frac{7.68\times 10^{10}}{2.04\times 10^4}\approx 3.8\times 10^6$ — millions of times faster. This is why ANN exists: exact scan is $O(Nd)$, ANN is ~$O(\log N)$.

Order: Chunk documents → Embed chunks → Embed the user query → Search top-k → Feed chunks to LLM.

• Chunk first: you can't embed a "focused idea" until you've split into focused passages (Chunking Strategies).
• Embed chunks: turns each passage into a searchable arrow; store vector + original text.
• Embed query with the SAME model: so query and chunks share one coordinate space.
• Search top-k: geometric nearest-neighbour lookup.
• Feed to LLM: this final step is Retrieval-Augmented Generation — retrieved text is injected into the prompt (Prompt Engineering). Mnemonic: "Chess Every Query Sees Rankings."

Geometric picture: one 2,000-token chunk = one averaged arrow pointing toward the "center of mass" of five different topics. That average direction matches no single query well, so cosine to any specific question is lukewarm — the correct FAQ is diluted by four irrelevant ones. Fix: chunk to coherent passages (~100–300 tokens, roughly one FAQ each) with slight overlap, so each arrow points cleanly at one idea. See Chunking Strategies.

If $\Vert \vec{a}\Vert =\Vert \vec{b}\Vert =1$, then $\text{sim}=\frac{\vec{a}\cdot \vec{b}}{1\cdot 1}=\vec{a}\cdot \vec{b}$. So after a one-time normalization at insert time, every query does only a cheap dot product — no per-query square roots or divisions. Over billions of comparisons that's a large saving, and it's exactly what production indexes do.

Probability: the only failure mode is the true best chunk not being among the 50 ANN candidates — that's exactly the recall@50 miss, which happens $1-0.9=0.1$ of the time. So success $=0.9=90\%$. (Note this is recall@50, not recall@1 — we only need the true best to be somewhere in the 50, because the perfect reranker will then surface it.) Why fetch 50: ANN is fast but approximate; casting a wide net (50 candidates) makes it very likely the true best is somewhere in the set even if it isn't ranked #1 by ANN. The exact reranker is slow, so we only run it on 50 items, not on all $N$. This combines ANN's speed with exact scoring's accuracy — the best of both.

In high $d$, two random unit vectors are almost always nearly orthogonal, so $\cos \theta \approx 0$ and thus $\Vert \vec{a}-\vec{b}{\Vert }^2\approx 2$ for almost every pair. The distances (and cosines) bunch into a narrow band, so the gap between the true nearest neighbour and a mediocre one shrinks — exact search must compare tiny differences, and noise can flip the ranking. This is the Curse of Dimensionality; it's a big reason we lean on trained embeddings (which push meaningful pairs to distinctly higher cosine) plus ANN indexes.

(a) $\Vert \vec{q}\Vert =\sqrt{0+9+16}=5$, so $\hat{q}=(0,0.6,0.8)$. (b) $C_1$: dot $=0+18+32=50$; $\Vert C_1\Vert =\sqrt{0+36+64}=10$; sim $=\frac{50}{5\cdot 10}=1$. $C_2$: dot $=0+0+0=0$; sim $=0$. (c) Retrieve $C_1$. sim $=1$ → angle $0°$ ($\vec{q}$ and $C_1$ are the same direction, $C_1=2\vec{q}$). sim $=0$ → angle $90°$ ($C_2$ is orthogonal → semantically unrelated).