Exercises — Retrieval-Augmented Generation (RAG) architecture

• Indexing — offline, done once: chunk every document, turn each chunk into a vector, store the vectors.
• Retrieval — online: embed the query, score chunks, keep top-$k$.
• Generation — online: paste chunks into the prompt, LLM writes the answer.

Mnemonic from the parent: CERG = Chunk → Embed → Retrieve → Generate.

• $x$ = the query (what the user asked).
• $y$ = the answer the model writes.
• $z$ = a single retrieved passage (a chunk) — the thing we sum over because we don't know in advance which passage holds the answer.

The $p(z\mid x)$ factor is the retriever (how likely this chunk is chosen); $p(y\mid x,z)$ is the generator (how likely the answer given query + chunk).

Cosine formula: $\cos \theta =\frac{\mathbf{q}\cdot \mathbf{d}}{\Vert \mathbf{q}\Vert \Vert \mathbf{d}\Vert }$.

First $\Vert \mathbf{q}\Vert =\sqrt{3^2+4^2}=\sqrt{25}=5$.

Chunk A $=(6,8)$: dot $=3\cdot 6+4\cdot 8=18+32=50$. $\Vert {\mathbf{d}}_A\Vert =\sqrt{36+64}=10$. $\cos =\frac{50}{5\cdot 10}=\frac{50}{50}=1.$ Same direction — A is just $\mathbf{q}$ scaled up ×2. Look at the red arrow lying exactly along $\mathbf{q}$.

Chunk B $=(4,3)$: dot $=3\cdot 4+4\cdot 3=12+12=24$. $\Vert {\mathbf{d}}_B\Vert =\sqrt{16+9}=5$. $\cos =\frac{24}{5\cdot 5}=\frac{24}{25}=\mathrm{0.96.}$ Very close in angle, slightly off.

Chunk C $=(-3,-4)$: dot $=3(-3)+4(-4)=-9-16=-25$. $\Vert {\mathbf{d}}_C\Vert =5$. $\cos =\frac{-25}{5\cdot 5}=-1.$ Exactly opposite direction — maximally irrelevant.

Order: A (1.0) > B (0.96) > C (−1.0). Notice A won even though its numbers were biggest — cosine divided the magnitude out, exactly as the parent note promised.

Why exponentiate? Raw scores can be any real number (even negative). $e^s$ is always positive, and dividing by the total forces the outputs to sum to 1 — a valid probability distribution.

• $e^3=20.0855$, $e^1=2.7183$, $e^0=1$.
• Denominator $=20.0855+2.7183+1=23.8038$.
• $p(z_1)=20.0855/23.8038=0.8438$.
• $p(z_2)=2.7183/23.8038=0.1142$.
• $p(z_3)=1/23.8038=0.0420$.

Chunk 1 grabs ~84% of the probability mass. A score gap of just 2 became a probability that is ~7× larger — softmax is sharp, which is why summing over only the top-$k$ chunks throws away almost nothing.

(a) Raw dot product:

• $\mathbf{q}\cdot {\mathbf{d}}_A=1\cdot 10+0\cdot 1=10.$
• $\mathbf{q}\cdot {\mathbf{d}}_B=1\cdot 1+0\cdot 0=1.$
• Raw dot ranks A > B.

(b) Cosine:

• $\Vert {\mathbf{d}}_A\Vert =\sqrt{100+1}=\sqrt{101}\approx 10.0499$. ${\cos }_A=10/10.0499=\mathrm{0.99504.}$
• $\Vert {\mathbf{d}}_B\Vert =1$. ${\cos }_B=1/1=1.$
• Cosine ranks B > A.

(c) The disagreement: B points exactly along $\mathbf{q}$ (angle 0°, perfect topical match) but is a short vector. A points almost along $\mathbf{q}$ but is 10× longer, so its raw dot product is inflated by magnitude. Look at the figure: the amber arrow (B) sits on the axis; the cyan arrow (A) is longer but tilted off. Raw dot product rewarded length; cosine rewarded direction. Cosine is the honest topical judge. This is exactly the "raw dot product biases toward high-norm chunks" mistake from the parent.

• With $k=2$ you keep chunks 1–2: one relevant, one distractor. The relevant chunk sits at the top of the prompt where models attend well.
• With $k=5$ you keep all five: now three distractors sit in the middle of the prompt. Models suffer the lost-in-the-middle effect — they under-attend to mid-prompt text — so the answer buried at position 5 may be ignored, while a middle distractor misleads the model.
• Recall vs precision: more $k$ raises recall (higher chance the answer is somewhere in context) but lowers precision (fraction of context that is actually useful). Beyond a point the added distractors + longer context cost more accuracy than the extra recall buys.
• Fix: small $k$ (≈3–8) plus a re-ranker to push the truly relevant chunk to the top.

Apply the parent's decomposition directly: $p(y\mid x)=p(z_1\mid x)p(y\mid x,z_1)+p(z_2\mid x)p(y\mid x,z_2).$

• Term 1: $0.7\times 0.9=0.63$ — the good passage is likely retrieved and produces a good answer, so it contributes a big chunk of the total.
• Term 2: $0.3\times 0.2=0.06$ — the weak passage is less likely retrieved and produces poor answers, so it barely helps.
• Marginal $p(y\mid x)=0.63+0.06=\mathrm{0.69.}$

Reading it: the answer quality is a weighted average over passages, weighted by how likely each is retrieved. Retrieval quality ($0.7$ on the good passage) and generator grounding ($0.9$) multiply — improving either lifts the whole system. This is why the parent says "retrieval quality dominates".

Template:

Use ONLY the context to answer. If the context does not
contain the answer, reply exactly: "Not found in context."
Context: Orders over $50 ship free.
Q: Do you ship to Canada?
A:

Ideal behaviour: the model outputs "Not found in context." — because the retrieved chunk says nothing about Canada.

Which mistake this defends against: the parent's "RAG eliminates hallucination" trap. Grounding alone does not stop confabulation when retrieval is off-topic; you must instruct the model to abstain. Combine with measuring retrieval quality separately (recall@k). See Prompt Engineering and LLM Hallucination.

Choose (b) RAG. Reasons:

1. Freshness without retraining — update a document in the vector store and the system is instantly correct; fine-tuning nightly is slow, costly, and always a day stale. Facts become data, not parameters.
2. Provenance — RAG can cite the exact chunk it used; a fine-tuned model's knowledge is diffused across weights with no citation. See Fine-tuning vs In-context Learning.
3. Lower hallucination on volatile facts — prices/stock in the prompt are copied, not recalled from foggy memory.

Failure mode of RAG: if retrieval returns an outdated or wrong chunk (e.g. stale index), the model confidently repeats it. Mitigation: keep the index refresh tight, measure recall@k, add a re-ranker, and instruct the model to say "Not found" when context is thin.

(a) Chunks per document. With stride 350 over 5000 tokens: number of chunks $=\lceil 5000/350\rceil =\lceil \mathrm{14.28...}\rceil =15$ chunks per document. Total $=15\times 1,000,000=15,000,000$ chunks.

(b) Index size. Each vector $=768\times 4=3072$ bytes. Total bytes $=15,000,000\times 3072=4.608\times 10^{10}$ bytes $=46.08$ GB (raw vectors, before index overhead).

(c) Why not one giant chunk per doc? A 5000-token chunk produces a diluted embedding — one vector must represent many unrelated sub-topics, so cosine matching becomes vague and the right passage never rises to the top. It also wastes the LLM's context window and drowns the answer among irrelevant text. Chunking exists precisely to keep each vector topically focused. See Chunking Strategies.

Order the five levels by what they test ::: L1 recognise pieces, L2 apply cosine/softmax, L3 analyse ranking/context tradeoffs, L4 synthesise the full probability + prompt, L5 design a whole system. The single biggest lever on RAG quality ::: retrieval quality (good chunking + embeddings + top-k re-ranking), not the LLM.