Worked examples — Retrieval-Augmented Generation (RAG) architecture

This page is a drill sheet. The parent note (RAG architecture) built the machinery: embeddings, cosine similarity, the retrieval softmax, and the prompt-assembly step. Here we push every button — every sign of cosine, the degenerate zero-vector case, the limiting behaviours of softmax, a real word problem, and an exam twist — so no scenario surprises you.

Before we start, one reminder built from zero.

---

The scenario matrix

Every RAG numeric task lands in one of these cells. The examples below fill all of them.

#	Case class	What is weird about it	Example
1	Same-direction chunk (best case)	$\cos \theta =+1$, angle $=0°$	Ex 1
2	Partially-on-topic	$0<\cos \theta <1$, e.g. $45°$	Ex 1
3	Orthogonal / off-topic	$\cos \theta =0$, angle $=90°$	Ex 2
4	Opposite direction (contradiction)	$\cos \theta <0$, angle up to $180°$	Ex 2
5	Magnitude trap (long vs short chunk)	dot product misleads, cosine fixes	Ex 3
6	Zero-vector degenerate input	division by $\Vert 0\Vert =0$	Ex 4
7	Softmax over scores (top-$k$ weighting)	turns scores into probabilities	Ex 5
8	Softmax limiting behaviour	equal scores → uniform; huge gap → one-hot	Ex 6
9	Real-world word problem (end-to-end)	pick $k$, assemble prompt, latency	Ex 7
10	Exam twist (normalisation equivalence)	why dot = cosine after normalising	Ex 8

---

Example 1 — best case and partial case (cells 1 & 2)

Figure 1 below draws all four arrows from the origin. Look at the two arrows lying flat along the horizontal axis — the navy query $\mathbf{q}$ and magenta $A$ overlap in direction; that overlap is what "cosine $=1$" looks like. The violet arrow $B$ splits the corner at $45°$, and the orange arrow $C$ shoots straight up, a full right angle away.

Figure 1. Query and three chunks. Navy = $\mathbf{q}$, magenta = $A$ (same direction, sim 1), violet = $B$ (45°, sim 0.707), orange = $C$ (90°, sim 0).

1. Score A. $\mathrm{sim}(\mathbf{q},A)=\frac{(1)(3)+(0)(0)}{1\cdot 3}=\frac{3}{3}=1$. Why this step? $A$ points along the same ray as $\mathbf{q}$ (angle $0°$), so cosine hits its maximum $+1$ — cell 1. In Figure 1 this is the magenta arrow lying on top of the navy one.
2. Score B. $\mathrm{sim}(\mathbf{q},B)=\frac{(1)(1)+(0)(1)}{1\cdot \sqrt{2}}=\frac{1}{\sqrt{2}}\approx 0.707$. Why this step? $B$ sits at $45°$ from the query — half on-topic — cell 2 (the violet arrow in Figure 1).
3. Score C. $\mathrm{sim}(\mathbf{q},C)=\frac{(1)(0)+(0)(5)}{1\cdot 5}=\frac{0}{5}=0$. Why this step? $C$ points straight up, $90°$ from $\mathbf{q}$ — perpendicular, no topical overlap (the orange arrow in Figure 1; previews cell 3).
4. Rank: $A>B>C$. Retrieve $A$.

Recall Verify

$A$'s numbers are 3× bigger than the query yet cosine stayed exactly 1 — the length divided out. If you had used the raw dot product you'd still pick $A$ here, but for the wrong reason (magnitude). That trap is Example 3.

---

Example 2 — orthogonal and opposite (cells 3 & 4)

In Figure 2, notice the dashed navy arc sweeping from the query all the way round to $E$: that is the $180°$ half-turn. The violet arrow $D$ standing at a right angle is the "$\mathrm{sim}=0$" picture; the magenta arrow $E$ pointing backwards is the "$\mathrm{sim}=-1$" picture. Direction, not length, is doing all the work.

Figure 2. Same query $\mathbf{q}$ (navy). Violet $D$ at 90° gives sim 0; magenta $E$ pointing backwards at 180° gives sim −1. The dashed arc is the half-turn from $\mathbf{q}$ to $E$.

1. Score D. $\mathrm{sim}(\mathbf{q},D)=\frac{(1)(0)+(0)(1)}{1\cdot 1}=0$. Why? Angle is exactly $90°$; $\cos 90°=0$. "Neither for nor against" — cell 3 (violet arrow in Figure 2).
2. Score E. $\mathrm{sim}(\mathbf{q},E)=\frac{(1)(-2)+(0)(0)}{1\cdot 2}=\frac{-2}{2}=-1$. Why? $E$ points the opposite way ($180°$). Cosine reaches its minimum $-1$ — cell 4 (magenta arrow in Figure 2). A negative score means "points away from the query's topic."
3. Ranking meaning. In nearest-neighbour search we keep the highest cosine, so $D$ (0) ranks above $E$ ($-1$). Both are bad, but $E$ is worse.

Recall Verify

Range check: cosine is always in $[-1,1]$ because $|\mathbf{a}\cdot \mathbf{b}|\le \Vert \mathbf{a}\Vert \Vert \mathbf{b}\Vert$ (Cauchy–Schwarz). Our values $0$ and $-1$ sit inside — sane. A cosine of $-1$ is the most anti-relevant a chunk can be.

---

Example 3 — the magnitude trap (cell 5)

1. Raw dot products. $\mathbf{q}\cdot S=(1)(1)+(1)(1)=2$. $\mathbf{q}\cdot L=(1)(10)+(1)(0)=10$. Why this step? Un-normalised dot product is inflated by length; $L$ wins only because its numbers are big — wrong.
2. Cosine of S. $\mathrm{sim}(\mathbf{q},S)=\frac{2}{\sqrt{2}\cdot \sqrt{2}}=\frac{2}{2}=1$. Why? Same direction as query → perfect topical match.
3. Cosine of L. $\mathrm{sim}(\mathbf{q},L)=\frac{10}{\sqrt{2}\cdot 10}=\frac{10}{10\sqrt{2}}=\frac{1}{\sqrt{2}}\approx 0.707$. Why? $L$ is $45°$ off; dividing by its length $10$ cancels its unfair size.
4. Verdict. Dot product ranks $L>S$ (wrong). Cosine ranks $S>L$ (right). Retrieve $S$.

Recall Verify

Sanity: $S$ is literally identical in direction to the query, so nothing should beat it. Cosine agrees ($1.0$). This is the whole reason production stores normalise vectors once at index time — see Example 8.

---

Example 4 — the zero-vector degenerate case (cell 6)

1. Norm of Z. $\Vert Z\Vert =\sqrt{0^2+0^2}=0$. Why this step? Cosine divides by $\Vert Z\Vert$, so we must check it first.
2. The formula breaks. $\mathrm{sim}(\mathbf{q},Z)=\frac{\mathbf{q}\cdot Z}{\Vert \mathbf{q}\Vert \Vert Z\Vert }=\frac{0}{1\cdot 0}=\frac{0}{0}$ — indeterminate (both numerator and denominator are $0$). Why? A zero vector has no direction, so "angle between them" is meaningless — there is no $\theta$ to take a cosine of.
3. Engineering fix. Systems either (a) drop zero-norm chunks at index time, or (b) define the similarity as $0$ so they rank last. Never let $0/0$ reach production.

Recall Verify

Indeterminate, not infinite: here both $\mathbf{q}\cdot Z=0$ and $\Vert Z\Vert =0$, so we get $0/0$, which has no value — you cannot decide it by the formula alone. And the limit is not fixed either: if a chunk shrinks toward $0$ along a direction at angle $\theta$ to $\mathbf{q}$, $\mathrm{sim}$ stays equal to $\cos \theta$ the whole way (length cancels top and bottom), so the limiting value depends on the approach direction — different directions give different answers. That path-dependence is exactly why $Z=0$ has no well-defined similarity and must be special-cased.

---

Example 5 — softmax turns scores into retrieval weights (cell 7)

1. Exponentiate each score. $e^4=54.598$, $e^2=7.389$, $e^1=2.718$. Why this step? Scores can be negative; $e^s$ is always positive, and it magnifies gaps so the best chunk stands out.
2. Sum them (over all $N=3$ chunks). $54.598+7.389+2.718=64.705$. Why? This normaliser — the denominator running over every chunk $j=1,2,3$ — forces the outputs to add to exactly $1$, a valid probability distribution.
3. Divide. $p(z_1\mid x)=54.598/64.705\approx 0.844$, $p(z_2\mid x)=7.389/64.705\approx 0.114$, $p(z_3\mid x)=2.718/64.705\approx 0.042$. Why? A score gap of $2$ became a probability ratio of $\approx 7.4\times$ — softmax is sharp.
4. Interpretation. The generator is conditioned mostly on chunk 1, so summing over just the top-$k$ (here top-1 or top-2) loses almost nothing.

Recall Verify

$0.844+0.114+0.042=1.000$ ✓ — a proper distribution. This is why summing over only the top-$k$ passages (rather than the whole corpus) barely changes the answer: the tail probabilities are tiny.

---

Example 6 — softmax limits and temperature (cell 8)

Figure 3 shows the same scores $s=[4,2,1]$ pushed through three temperatures, so you can see the distribution slide from flat to peaked.

Figure 3. Softmax of $s=[4,2,1]$ at $T=5$ (flat, magenta), $T=1$ (plain, violet), $T=0.3$ (peaked, orange). Lower temperature concentrates the probability on chunk 1.

1. Case (a): equal scores. $p_i=\frac{e^3}{3e^3}=\frac{1}{3}\approx 0.333$ each. Why? Identical scores cancel inside the fraction → uniform. The retriever is maximally unsure; every chunk equally likely — the same shape a very high temperature would force.
2. Case (b): one huge score. $e^{100}$ dwarfs $e^1$, so $p_1\approx 1$ and $p_2,p_3\approx 0$. Why? As the gap $\to \infty$, softmax approaches a one-hot vector — retrieval collapses to a single confident chunk, the same shape a very low temperature would force.
3. Takeaway. Whether from the scores themselves (these two cases) or from the temperature dial (Figure 3), softmax slides between "hedge across everything" (flat/uniform) and "bet on one" (peaked/one-hot).

Recall Verify

Case (a): $3\times \frac{1}{3}=1$ ✓. Case (b): $p_1=\frac{e^{100}}{e^{100}+2e}\approx 1$ to 40+ decimal places — confirming the one-hot limit. Both extremes are valid probability distributions.

---

Example 7 — real-world word problem, end to end (cell 9)

1. Choose $k$. There is a clear score cliff between chunk 2 ($0.79$) and chunk 3 ($0.31$). Keep $k=2$. Why this step? Chunks 3–5 are near-irrelevant distractors. Stuffing them in risks the "lost-in-the-middle" effect (models skim over mid-prompt text) and wastes tokens — a small, precise $k$ beats a big noisy one.
2. Assemble the prompt. SYSTEM: Answer ONLY from context; say "not found" if absent. CONTEXT: <chunk 1> <chunk 2> QUESTION: How long is the refund window? ANSWER: Why? Putting the fact in the prompt lets the model copy, not invent → grounded answer "30 days" with a citation.
3. Latency of the extra 3 chunks. Tokens saved $=3\times 120=360$. Cost $=360\times \frac{3\text{ ms}}{100}=10.8$ ms. Why? $360/100=3.6$ blocks, each $3$ ms $\Rightarrow 3.6\times 3=10.8$ ms saved and less distraction. Win–win.

Recall Verify

$3\times 120=360$ tokens; $360/100\times 3=10.8$ ms. Small in absolute terms, but multiplied across millions of queries it matters — and the quality gain from dropping distractors is the bigger prize, which is why re-ranking a small candidate set is so effective.

---

Example 8 — exam twist: dot product = cosine after normalising (cell 10)

1. Norm of d. $\Vert \mathbf{d}\Vert =\sqrt{3^2+4^2}=\sqrt{25}=5$. Why this step? Normalising means scaling to length 1; we need the length first. (Recall the hat: $\hat{\mathbf{d}}=\mathbf{d}/\Vert \mathbf{d}\Vert$.)
2. Normalise. $\hat{\mathbf{d}}=(3/5,4/5)=(0.6,0.8)$. And $\hat{\mathbf{q}}=(0,1)$ already, since $\Vert \mathbf{q}\Vert =1$. Why? Dividing by the norm keeps the direction, sets the length to 1.
3. Dot of normalised vectors. $\hat{\mathbf{q}}\cdot \hat{\mathbf{d}}=(0)(0.6)+(1)(0.8)=0.8$. Why? When both lengths are 1, the two $\Vert \cdot \Vert$ factors in the denominator of cosine are both $1$, so the cosine formula collapses into just the dot product.
4. Cosine of originals. $\mathrm{sim}(\mathbf{q},\mathbf{d})=\frac{\mathbf{q}\cdot \mathbf{d}}{\Vert \mathbf{q}\Vert \Vert \mathbf{d}\Vert }=\frac{(0)(3)+(1)(4)}{1\cdot 5}=\frac{4}{5}=0.8$. Identical to step 3.
5. The shortcut, stated. Because $\mathrm{sim}(\mathbf{q},\mathbf{d})=\frac{\mathbf{q}}{\Vert \mathbf{q}\Vert }\cdot \frac{\mathbf{d}}{\Vert \mathbf{d}\Vert }=\hat{\mathbf{q}}\cdot \hat{\mathbf{d}}$ always (not just for these numbers), you can normalise every vector once at index time and then rank by the cheaper dot product — the ranking is provably identical to cosine.

Recall Verify

Both routes give $0.8$ ✓. This is why systems normalise once at index time (Phase A) and then use fast dot-product nearest-neighbour search — same ranking as cosine, less compute. Contrast Example 3, where skipping normalisation broke the ranking.

---