Retrieval-Augmented Generation (RAG) architecture

WHY does RAG exist? (the problem it solves)

An LLM's knowledge is frozen at its training cutoff and compressed into billions of weights. This creates three pains:

Staleness — it can't know events after training.
Hallucination — when unsure, it fabricates fluent-but-wrong facts.
No provenance — you can't cite where an answer came from.

HOW it works — the full pipeline

Figure — Retrieval-Augmented Generation (RAG) architecture

Phase A — Indexing (offline, done once)

Chunk documents into passages (e.g. ~200–500 tokens).
Embed each chunk with an encoder $E_d(\cdot)$ into a vector $\mathbf{d}_i \in \mathbb{R}^n$ .
Store vectors in a vector database (FAISS, Pinecone…) for fast nearest-neighbour search.

Phase B — Retrieval (online, per query)

Embed the query: $\mathbf{q} = E_q(x)$ .
Score every chunk by similarity, usually cosine: $\text{sim}(\mathbf{q},\mathbf{d}_i)=\frac{\mathbf{q}\cdot\mathbf{d}_i}{\|\mathbf{q}\|\,\|\mathbf{d}_i\|}$
Take the top- $k$ highest-scoring chunks.

Phase C — Generation (online)

Build a prompt: SYSTEM + [retrieved chunks] + QUESTION.
LLM generates the answer grounded in those chunks (and ideally cites them).

Deriving the retrieval objective from first principles

Formula Why cosine similarity, step by step

Goal: find the chunk most "about the same thing" as the query.

Why start with a dot product? $\mathbf{q}\cdot\mathbf{d} = \|\mathbf{q}\|\|\mathbf{d}\|\cos\theta$ . The dot product mixes direction (topic) and magnitude (vector length).
Why is magnitude bad here? A long document produces a large-norm vector and would win purely for being long, not relevant. We want topic, not length.
How to remove magnitude? Divide it out: $\cos\theta = \frac{\mathbf{q}\cdot\mathbf{d}}{\|\mathbf{q}\|\,\|\mathbf{d}\|}\in[-1,1]$
Now $\cos\theta = 1$ ⇒ identical direction ⇒ maximally relevant. This is exactly what we rank by. If vectors are pre-normalised ( $\|\mathbf{d}\|=1$ ), cosine equals the dot product — that's why production systems normalise once at index time and use fast dot-product search.

Common mistakes (steel-manned)

The 80/20 — what actually moves the needle

Retrieval quality dominates. A great LLM on bad chunks < an average LLM on the right chunks.
Three levers give ~80% of gains: good chunking, good embeddings/similarity, and ==top- $k$ + re-ranking==.

Recall Feynman: explain to a 12-year-old

Imagine an open-book exam. Instead of trying to remember everything, you first flip to the right page in your notebook, then write the answer using that page. RAG does the same for a chatbot: it finds the right page (retrieval) from a big pile of notes, tapes it next to the question, and then the chatbot answers by reading it. That way it doesn't make stuff up, and if you add a new page to the pile, it instantly knows the new thing — no need to send the chatbot back to school.

Flashcards

#flashcards/ai-ml

What problem does RAG primarily solve in LLMs?

Stale knowledge, hallucination, and lack of source provenance — by grounding answers in externally retrieved documents.

Write the RAG probabilistic decomposition.

p(y\mid x)=\sum_z p(z\mid x)\,p(y\mid x,z)

— retriever times generator, summed over latent passages.

Why use cosine similarity instead of raw dot product for retrieval?

Cosine removes vector magnitude, so ranking depends on topical direction, not document length — unless vectors are pre-normalised (then dot product is equivalent).

What are the three phases of a RAG pipeline?

Indexing (chunk+embed+store, offline), Retrieval (embed query, top-k nearest), Generation (LLM answers conditioned on retrieved context).

Why is top-k a valid approximation of the full sum over passages?

Softmax over similarities makes

p(z\mid x)

negligible for low-scoring chunks, so only the top few contribute meaningfully.

Does RAG fully eliminate hallucination?

No — if retrieval returns irrelevant/empty chunks the model can still confabulate; retrieval quality must be measured separately.

What is the 'lost-in-the-middle' effect and its RAG implication?

LLMs attend poorly to text in the middle of long contexts, so blindly increasing k hurts; keep k small and re-rank.

Why prefer RAG over fine-tuning for factual knowledge?

Facts stay as swappable external data; updating a document requires no retraining, unlike weights baked in by fine-tuning.

Softmax retrieval probability formula?

p(z_i\mid x)=e^{s_i}/\sum_j e^{s_j}

over similarity scores

s_i

What is the CERG mnemonic?

Chunk → Embed → Retrieve → Generate.

Connections

Vector Embeddings — how text becomes $\mathbf{d}_i$ ; the quality ceiling of retrieval.
Cosine Similarity & Nearest Neighbour Search — the ranking maths (FAISS/ANN).
Prompt Engineering — how retrieved chunks are injected + grounding instructions.
LLM Hallucination — the failure mode RAG mitigates.
Fine-tuning vs In-context Learning — parameters vs data trade-off.
Re-ranking & Cross-Encoders — precision boost after initial retrieval.
Chunking Strategies — 80/20 lever for retrieval quality.

Concept Map

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, ek normal LLM ek closed-book student ki tarah hai — jo training ke time yaad kiya bas wahi jaanta hai. Problem yeh hai ki uska gyaan purana ho jaata hai, aur jab confidence nahi hota toh woh confidently galat cheez bana deta hai (hallucination). RAG ka idea simple hai: pehle question ke hisaab se relevant documents dhoondo (retrieve karo), unko prompt me daal do, phir model unhi ko padh kar answer de. Matlab open-book exam! Isse facts model ke andar (weights me) store nahi hote, balki bahar ek database me rehte hai — naya document add karo, system turant smart ho jaata hai, bina dobara train kiye.

Pipeline yaad rakhne ke liye mnemonic hai CERG: pehle documents ko Chunk karo (chhote pieces), har chunk ko Embed karo (vector bana do), phir query aane par top-k similar chunks Retrieve karo, aur end me LLM se Generate karwao. Retrieval me hum cosine similarity use karte hai — kyunki hum topic match chahte hai, document ki length nahi. Dot product magnitude ko mila deta hai, isliye normalise karke sirf direction (angle) dekhte hai. Cosine = 1 matlab bilkul same topic.

Sabse important baat — 80/20 rule: retrieval quality hi sab kuch hai. Best LLM bhi galat chunks par galat answer dega, aur average LLM bhi sahi chunks mile toh sahi answer dega. Isliye chunking, embeddings, aur top-k + re-ranking par mehnat karo. Aur yeh galatfehmi mat rakhna ki "zyada chunks do toh better" — bade context me model beech ka text ignore kar deta hai (lost-in-the-middle), cost badhti hai, aur distractor chunks confuse karte hai. Chhota k (3–8) rakho aur re-rank karo. Yahi asli RAG hai.

Retrieval-Augmented Generation (RAG) architecture

WHY does RAG exist? (the problem it solves)

WHAT is RAG? (definition)

HOW it works — the full pipeline

Phase A — Indexing (offline, done once)

Phase B — Retrieval (online, per query)

Phase C — Generation (online)

Deriving the retrieval objective from first principles

Worked examples

Common mistakes (steel-manned)

The 80/20 — what actually moves the needle

Flashcards

Connections

Concept Map

Hinglish (regional understanding)

Go deeper — visual, from zero

Test yourself — Alignment, Prompting & RAG

Connections