Retrieval-Augmented Generation (RAG) architecture: Visual walkthrough

We are answering one question, slowly: "Given a question, how does a machine decide which pieces of text are worth reading — and then how does it use them to answer?"

Step 1 — Turn words into arrows

WHAT. A computer cannot compare two sentences directly — it only compares numbers. So we hand each piece of text to an encoder, a trained function that spits out a fixed list of numbers.

We only draw 2 numbers (2D) so we can see it. Real embeddings have hundreds — the geometry is identical, just invisible. See Vector Embeddings for how the encoder is trained.

WHY. Direction is the trick. If "refund policy" and "return within 30 days" are about the same thing, their arrows should aim the same way, even if one arrow is longer.

PICTURE. Two arrows leaving the same point. The angle between them, $θ$ , is the whole game — small angle means "same topic".

$q$ ::: the query arrow — the question, turned into numbers.
$d$ ::: a document arrow — one candidate chunk of text.
$θ$ ::: the angle between them — small angle = similar meaning.

Step 2 — The dot product measures overlap, but it cheats

WHAT. There is one arithmetic operation that already knows about the angle between two arrows: the dot product. For arrows $q = (q_{1}, q_{2})$ and $d = (d_{1}, d_{2})$ it is:

$q \cdot d = right \times right q_{1} d_{1} + up \times up q_{2} d_{2}$

$q_{1} d_{1}$ ::: multiply the two rightward amounts.
$q_{2} d_{2}$ ::: multiply the two upward amounts, then add.

WHY this tool and not another? Because of a beautiful fact (proved with a right triangle — see Cosine Similarity & Nearest Neighbour Search):

$q \cdot d = length of q ∥ q ∥ length of d ∥ d ∥ angle factor cos θ$

The bars $∥ q ∥$ mean length of the arrow (Pythagoras: $q_{1}^{2} + q_{2}^{2}$ ). And $cos θ$ — the cosine — is just a dial that reads $1$ when the arrows point the same way ( $θ = 0$ ), $0$ when they are perpendicular ( $θ = 90°$ ), and $- 1$ when opposite ( $θ = 180°$ ).

The cheat: the dot product multiplies by the two lengths. A long document makes a long arrow, so it scores high just for being long — not for being relevant. That is exactly the wrong thing to reward.

PICTURE. Same short query arrow, two document arrows at the same angle but different lengths. The longer one gets a bigger dot product — unfairly.

Step 3 — Divide the length out → cosine similarity

WHAT. We want the angle only. So take the dot product and cancel the two lengths by dividing by them:

$cos θ = \frac{q \cdot d overlap}{both lengths ∥ q ∥ ∥ d ∥}$

numerator $q \cdot d$ ::: the raw overlap from Step 2.
denominator $∥ q ∥ ∥ d ∥$ ::: the two lengths we want gone.

WHY. Rearrange Step 2: since $q \cdot d = ∥ q ∥∥ d ∥ cos θ$ , dividing both sides by the lengths leaves pure $cos θ$ — length has vanished. This number is called the cosine similarity, and it always lands in $[- 1, 1]$ .

PICTURE. The unit circle. Every arrow is shrunk to length $1$ . Put the query along the horizontal axis; then $cos θ$ is exactly the projection of the document's unit arrow onto the query's direction — i.e. how much of $\hat{d}$ lies along $\hat{q}$ . (It looks horizontal here only because we lined the query up with the horizontal axis; the real quantity is "shadow onto the other arrow's direction", not onto any fixed axis.)

Step 4 — Rank all chunks, keep the top few

WHAT. Score the query against every stored chunk $d_{1}, d_{2}, \dots$ , sort the scores, and keep the best $k$ (say $k = 3$ ). This is nearest-neighbour search.

WHY. We do not know in advance which chunk holds the answer, so we test them all and trust the ranking. Keeping only the top $k$ throws away the obviously-irrelevant ones so the model isn't drowned in noise.

PICTURE. One query arrow, many document arrows fanned around it. The three closest in angle are highlighted; the far-off ones are greyed out.

Step 5 — Turn scores into probabilities with softmax

WHAT. The scores are just numbers on a line (some negative). To reason about "how likely is chunk $z$ the right one", we convert scores into probabilities: positive numbers that add up to $1$ . The tool is the softmax:

$p (z_{i} ∣ x) = \frac{e ^{s_{i}} make positive}{normalise to sum 1 j \sum e ^{s_{j}}}$

$s_{i}$ ::: the similarity score of chunk $i$ .
$e^{s_{i}}$ ::: raise $e \approx 2.718$ to that score — always positive, and bigger scores blow up faster.
$\sum_{j} e^{s_{j}}$ ::: add all of them so the whole set sums to exactly $1$ .

WHY exponentiate and not just divide by the sum? Two reasons: raw scores can be negative (you can't have a negative probability), and $e^{s}$ sharpens the winner — a small score lead becomes a big probability lead, so the top chunk dominates. That is precisely why "top- $k$ is a fine approximation": the also-rans carry almost no probability.

PICTURE. Bars: flat similarity scores on the left, then the same scores after $e^{(\cdot)}$ and normalising — one bar towers over the rest.

Step 6 — Sum over passages: the RAG equation

WHAT. We still don't know the correct passage, so we treat the passage $z$ as a hidden (latent) choice and average over all choices, weighting each by how likely it was picked. This is the law of total probability:

$p (y ∣ x) = \sum_{z} Step 5: retriever picks z p (z ∣ x) generator writes y having read z p (y ∣ x, z)$

$x$ ::: the query (question).
$y$ ::: the answer we want.
$z$ ::: one retrieved passage — the hidden variable.
$p (z ∣ x)$ ::: chance the retriever hands us $z$ (the softmax from Step 5).
$p (y ∣ x, z)$ ::: chance the LLM writes answer $y$ after reading $x$ and $z$ .

WHY a sum? Because "the right passage" is uncertain. Instead of betting on one, we spread across all of them, each vote weighted by its retrieval probability. In practice we sum only over the top- $k$ (Step 4) — using the renormalised weights from Step 5 — because everything else has near-zero weight.

PICTURE. A tree: query at the root, branching to candidate passages $z_{1}, z_{2}, z_{3}$ (branch thickness = $p (z ∣ x)$ ), each leading to an answer, all merging into the final $p (y ∣ x)$ .

Step 7 — Edge & degenerate cases (never get surprised)

WHAT / WHY / PICTURE, all the corners the formula must survive:

Common mistake The corner cases

Zero vector ( $d = (0, 0)$ ): length is $0$ , so $∥ d ∥ = 0$ and cosine divides by zero — undefined. An empty/garbage chunk has no direction. Fix: filter empty chunks before indexing.
Opposite meaning ( $θ = 180°$ ): $cos θ = - 1$ , the lowest possible score → ranked last. But note: its softmax weight is $e^{- 1} > 0$ , not zero — so it is only excluded if you keep a small top- $k$ or apply a similarity threshold (a hard cutoff). Without such a cutoff, even anti-relevant text keeps a tiny nonzero weight.
All chunks equally bad (scores all $\approx$ equal): softmax spreads probability flatly → no clear winner. The system should answer "not found" rather than confabulate — see LLM Hallucination.
Long document ≠ relevant (Step 2's cheat): cosine cancels length, so a giant off-topic wall of text still scores low. Verified by the A vs B example.
Huge $k$ : sum in Step 6 barely changes (extra terms have tiny $p (z ∣ x)$ ) but the prompt gets bloated → lost-in-the-middle and distractors. Small $k$ + Re-ranking & Cross-Encoders wins.

The panels show, left to right: a zero-length arrow (no direction), an opposite arrow ( $cos = - 1$ , still tiny nonzero weight), and a fan of equal-angle arrows (flat softmax → "not found").

The one-picture summary

One diagram compresses the whole chain: words → arrows → cosine → top- $k$ (renormalised) → softmax weights → weighted sum = the RAG equation. Trace the arrows once and you have re-derived retrieval end to end.

Recall Feynman retelling — the whole walkthrough in plain words

You have a question and a big pile of notes. A machine can't read, so it turns every note and your question into an arrow made of numbers — arrows about the same topic point the same way. To measure "same way", it tries multiplying the arrows (the dot product), but that secretly rewards long notes just for being long. So it divides that length back out, leaving a pure "how aligned are we" score between $- 1$ and $1$ — the cosine, which we name $sim (q, d)$ . It scores every note this way, keeps the best three, and then softens those scores into confidences that add up to one (softmax, renormalised over just the kept notes), where the best note grabs almost all the confidence. Finally it writes the answer by averaging over those top notes, each weighted by its confidence — that weighted average is the famous RAG equation. And it stays honest at the edges: an empty note has no direction (skip it), an opposite note scores lowest (drop it with a cutoff), and if nothing looks good it should just say "I couldn't find it" instead of making something up.

Recall Quick self-check

Why does cosine beat the raw dot product? ::: It divides out both arrow lengths, so ranking depends on direction (topic), not document length — unless arrows are pre-normalised, when the two are equal. What does $sim (q, d)$ mean? ::: It is just a name for $cos θ$ , the cosine of the angle between the query and document arrows. Why exponentiate scores in softmax? ::: To make every score positive and force them to sum to 1, while sharpening the lead so the top chunk dominates — justifying the top- $k$ shortcut. After truncating to top- $k$ , what must you do to the probabilities? ::: Renormalise (divide by the sum of the kept ones) so they sum back to 1 over just those $k$ passages. What is $z$ in $p (y ∣ x) = \sum_{z} p (z ∣ x) p (y ∣ x, z)$ ? ::: A latent (hidden) retrieved passage; we sum because we don't know which single passage is correct. What breaks cosine, and how to fix it? ::: A zero vector (empty chunk) → divide-by-zero; filter empty chunks before indexing.