Retrieval-Augmented Generation (RAG) architecture

4.4.11 · AI-ML › Alignment, Prompting & RAG

RAG kyun exist karta hai? (wo problem jise ye solve karta hai)

Ek LLM ki knowledge uski training cutoff pe frozen hoti hai aur billions of weights mein compress hoti hai. Isse teen problems paida hoti hain:

Staleness — training ke baad ki events ke baare mein ye kuch nahi jaanta.
Hallucination — jab unsure hota hai, to fluent-but-wrong facts fabricate kar deta hai.
No provenance — aap cite nahi kar sakte ki jawab kahan se aaya.

Ye kaise kaam karta hai — poora pipeline

Figure — Retrieval-Augmented Generation (RAG) architecture

Phase A — Indexing (offline, ek baar hota hai)

Documents ko passages mein Chunk karo (jaise ~200–500 tokens).
Har chunk ko ek encoder $E_{d} (\cdot)$ se Embed karo ek vector $d_{i} \in R^{n}$ mein.
Vectors ko ek vector database (FAISS, Pinecone…) mein Store karo fast nearest-neighbour search ke liye.

Phase B — Retrieval (online, har query pe)

Query ko embed karo: $q = E_{q} (x)$ .
Har chunk ko similarity se score karo, usually cosine: $sim (q, d_{i}) = \frac{q \cdot d _{i}}{∥ q ∥ ∥ d _{i} ∥}$
Top- $k$ highest-scoring chunks lo.

Phase C — Generation (online)

Ek prompt banao: SYSTEM + [retrieved chunks] + QUESTION.
LLM un chunks pe grounded hokar jawab generate karta hai (aur ideally unhe cite bhi karta hai).

Retrieval objective ko first principles se derive karna

Formula Cosine similarity kyun, step by step

Goal: wo chunk dhundho jo query ke saath "same cheez ke baare mein" ho.

Dot product se shuru kyun karein? $q \cdot d = ∥ q ∥∥ d ∥ cos θ$ . Dot product direction (topic) aur magnitude (vector length) dono ko mix karta hai.
Yahan magnitude buri kyun hai? Ek lamba document ek large-norm vector produce karta hai aur sirf lamba hone ki wajah se jeet jaata, relevant hone ki wajah se nahi. Hume topic chahiye, length nahi.
Magnitude kaise hatayein? Use divide kar do: $cos θ = \frac{q \cdot d}{∥ q ∥ ∥ d ∥} \in [- 1, 1]$
Ab $cos θ = 1$ ⇒ identical direction ⇒ maximally relevant. Hum exactly isi se rank karte hain. Agar vectors pre-normalised hain ( $∥ d ∥ = 1$ ), to cosine dot product ke barabar hota hai — isliye production systems index time pe ek baar normalise karte hain aur fast dot-product search use karte hain.

Common mistakes (steel-manned)

The 80/20 — kya cheez actually kaam karti hai

Retrieval quality dominate karti hai. Bure chunks pe great LLM < sahi chunks pe average LLM.
Teen levers ~80% gains dete hain: good chunking, good embeddings/similarity, aur ==top- $k$ + re-ranking==.

Recall Feynman: ek 12-saal ke bachche ko explain karo

Soch ek open-book exam. Saab kuch yaad karne ki koshish karne ki jagah, pehle apni notebook mein sahi page pe flip karo, phir us page ko use karke jawab likho. RAG ek chatbot ke liye wohi karta hai: wo notes ke ek bade dher mein se sahi page dhundta hai (retrieval), usse sawaal ke paas tape karta hai, aur phir chatbot use padhke jawab deta hai. Is tarah wo cheezein banata nahi, aur agar tum dher mein ek naya page add karo, to use turant naya knowledge ho jaata hai — chatbot ko wapas school bhejne ki zaroorat nahi.

Flashcards

#flashcards/ai-ml

What problem does RAG primarily solve in LLMs?

Stale knowledge, hallucination, aur source provenance ka na hona — externally retrieved documents mein answers ground karke.

Write the RAG probabilistic decomposition.

p (y ∣ x) = \sum_{z} p (z ∣ x) p (y ∣ x, z)

— retriever times generator, latent passages pe summed.

Why use cosine similarity instead of raw dot product for retrieval?

Cosine vector magnitude hatata hai, isliye ranking topical direction pe depend karti hai, document length pe nahi — jab tak vectors pre-normalised na hon (tab dot product equivalent hai).

What are the three phases of a RAG pipeline?

Indexing (chunk+embed+store, offline), Retrieval (embed query, top-k nearest), Generation (LLM retrieved context pe conditioned hokar jawab deta hai).

Why is top-k a valid approximation of the full sum over passages?

Similarities pe softmax low-scoring chunks ke liye

p (z ∣ x)

negligible bana deta hai, isliye sirf top kuch hi meaningfully contribute karte hain.

Does RAG fully eliminate hallucination?

Nahi — agar retrieval irrelevant/empty chunks return kare to model phir bhi confabulate kar sakta hai; retrieval quality alag se measure karni chahiye.

What is the 'lost-in-the-middle' effect and its RAG implication?

LLMs lambe contexts ke middle mein text ko poorly attend karte hain, isliye blindly k badhana hurt karta hai; k chhota rakho aur re-rank karo.

Why prefer RAG over fine-tuning for factual knowledge?

Facts swappable external data rehte hain; ek document update karne ke liye koi retraining nahi chahiye, unlike fine-tuning se weights mein bake kiye facts.

Softmax retrieval probability formula?

p (z_{i} ∣ x) = e^{s_{i}} / \sum_{j} e^{s_{j}}

similarity scores

s_{i}

pe.

What is the CERG mnemonic?

Chunk → Embed → Retrieve → Generate.

Connections

Vector Embeddings — text $d_{i}$ kaise banta hai; retrieval ki quality ceiling.
Cosine Similarity & Nearest Neighbour Search — ranking ka maths (FAISS/ANN).
Prompt Engineering — retrieved chunks kaise inject hote hain + grounding instructions.
LLM Hallucination — wo failure mode jise RAG mitigate karta hai.
Fine-tuning vs In-context Learning — parameters vs data trade-off.
Re-ranking & Cross-Encoders — initial retrieval ke baad precision boost.
Chunking Strategies — retrieval quality ke liye 80/20 lever.