Reranking and hybrid search — Hinglish notes

4.4.14 · AI-ML › Alignment, Prompting & RAG

1. Ye cheez chahiye hi kyun?

Ek naive RAG pipeline kuch is tarah kaam karta hai: embed query → nearest neighbours in vector DB → feed top-k to LLM. Do failure modes hain:

Dense (vector) search exact terms miss karta hai. Embeddings meaning capture karte hain, isliye "ERR_4092" error code ya "Metformin" drug ki query ke liye semantically "nearby" lekin literally galat chunks retrieve ho sakte hain. Isme exact keyword match ka koi concept nahi hota.
Sparse (keyword) search meaning miss karta hai. "car" ki search us doc ko match nahi karegi jo sirf "automobile" kehti hai.

80/20 insight: retrieval quality — prompt tweaking nahi — usually RAG accuracy ka sabse bada lever hota hai. Hybrid + rerank aksar kisi bhi prompt engineering se behtar perform karta hai.

2. Do retrievers — first principles se derive kiye

2.1 Sparse: BM25 (KYA aur KYUN)

Hum score karna chahte hain ki document $D$ query ke terms $q_{1}, \dots, q_{n}$ ke liye kitna relevant hai. Ye intuitions build karne hain:

Jo term $D$ mein zyada baar aata hai woh score badhaye (term frequency, TF) — lekin diminishing returns ke saath (50th "cat" 5th se zyada kuch nahi batata).
Jo term poore corpus mein rare hai woh zyada discriminating hota hai (inverse document frequency, IDF). "The" bekaar hai; "photosynthesis" gold hai.
Lambe documents ko term-count ka naajayaz fayda milta hai, isliye hum length se normalise karte hain.

In teeno ideas ko formula mein daalte hain:

Ye shape kyun?

Fraction $\frac{f ( k _{1} + 1 )}{f + k _{1} ( \dots )}$ saturate karta hai: jaise $f \to \infty$ score $\to (k_{1} + 1) \cdot IDF$ ho jaata hai, diminishing returns deta hai. Yahi "50th cat" effect hai.
$k_{1}$ control karta hai kitni jaldi saturate hota hai; $b$ control karta hai kitni strongly length normalisation lagti hai ( $b = 0$ ⇒ length ignore karo, $b = 1$ ⇒ full normalisation).
$IDF (q_{i}) = ln \frac{N - n ( q _{i} ) + 0.5}{n ( q _{i} ) + 0.5} + 1$ , jahan $N$ = total docs, $n (q_{i})$ = $q_{i}$ wale docs — rare terms ko bada multiplier milta hai.

2.2 Dense: embeddings ki cosine similarity

Query aur doc ko vectors $q, d \in R^{m}$ mein embed karo ek bi-encoder se (har ek independently encode hota hai), score karo: $sim (q, d) = \frac{q \cdot d}{∥ q ∥ ∥ d ∥}$ Kyunki documents offline ek baar embed hote hain, yeh fast hai (approximate nearest neighbour search) — lekin query aur doc kabhi ek doosre ko "dekhte" nahi, isliye subtle interactions kho jaate hain.

3. Do lists merge karna — Reciprocal Rank Fusion (RRF)

BM25 aur cosine ke scores incomparable scales par hain. Hum unhe directly add nahi kar sakte. RRF is problem ko sidestep karta hai sirf har document ki rank use karke har list mein.

$1/ (k + rank)$ kyun? #1 rank wala document $\frac{1}{k + 1}$ contribute karta hai; #100 rank wala $\frac{1}{k + 100} \approx 0$ contribute karta hai. Toh top positions dominate karti hain, aur jo doc dono lists mein aata hai (chahe middling rank par) woh sirf ek mein aane wale doc se aage hota hai. Constant $k$ rank 1 aur rank 2 ke beech ka difference soften karta hai taaki winner-take-all na ho.

4. Cross-encoder se reranking (precision step)

Winning architecture: broadly aur saste mein retrieve karo (bi-encoder + BM25 → top-100), phir precisely aur expensively rerank karo (cross-encoder → top-5). Tum expensive cost sirf 100 items par pay karte ho, poore corpus par nahi.

$final top- k = TopK_{k} ({CrossEncoder (q, d) : d \in Hybrid-top- 100})$

5. Common mistakes (steel-manned)

6. Feynman

Recall Ek 12-saal ke bachche ko explain karo

Socho ek badi library mein koi fact dhundh rahe ho. Ek helper (keyword search) jo exact words tune bole unhe dhundha hai. Doosra helper (meaning search) unhi books ko dhundhta hai jo usi idea ke baare mein hain, chahe alag words se likha ho. Tum dono se poochte ho taaki kuch achha miss na ho — yahi hybrid search hai. Woh tumhe 100 books ki ek messy pile de dete hain. Phir ek bahut careful reader (the reranker) tumhara question aur har book saath mein actually padhta hai aur best 5 ko upar rakhta hai. Tum sirf woh 5 answer-writer ko dete ho, taaki woh ek badi messy pile se confuse na ho.

7. Flashcards

Pure vector search kya problem solve nahi kar sakta jo hybrid search karta hai?

Exact/rare keyword aur code matches (jaise "ERR_4092") jo embeddings miss karte hain; yeh lexical (BM25) recall ko semantic recall mein add karta hai.

Hybrid search mein do component retrievers kya hain?

Sparse/lexical (BM25 keyword) aur dense/semantic (embedding cosine similarity).

BM25 ka term-frequency term saturate kyun karta hai?

Diminishing returns — ek term ke extra occurrences kam info add karte hain; fraction

\frac{f ( k _{1} + 1 )}{f + k _{1} ( \dots )} \to (k_{1} + 1) IDF

jab

f \to \infty

BM25 parameters

k_{1}

aur

b

kya control karte hain?

k_{1}

= term-frequency kitni jaldi saturate hoti hai;

b

= document-length normalisation ki strength (

0

=none,

1

=full).

RRF formula kya hai?

RRF (d) = \sum_{r} \frac{1}{k + rank _{r} ( d )}

, typically

k = 60

Raw scores add karne ki jagah rank-based RRF kyun use karte hain?

BM25 aur cosine incomparable scales par hain; RRF sirf ranks use karta hai isliye scale-free hai aur dono retrievers se mile docs ko reward karta hai.

Bi-encoder aur cross-encoder mein kya farak hai?

Bi-encoder query aur doc alag embed karta hai (fast, precomputable, coarse); cross-encoder unhe jointly cross-attention se process karta hai (slow, accurate, per-pair).

Cross-encoder poore corpus search karne ke liye kyun use nahi ho sakta?

Isko per document ek forward pass chahiye aur embeddings precompute nahi kar sakta, isliye per query millions of docs score karna infeasible hai — yeh ek small candidate set par reranker hai.

Hybrid search kaun sa metric improve karta hai, aur reranking kaun sa?

Hybrid → recall (net chaura karo); reranking → precision@top-k (best ko pehle order karo).

LLM ko reranking ki jagah bada top-k feed kyun nahi karte?

Lost-in-the-middle: dabe relevant chunks ignore ho jaate hain, plus zyada cost, latency, aur noise se hallucination badhta hai.

Typical two-stage retrieve-then-rerank sizing kya hoti hai?

~100 candidates saste mein retrieve karo (hybrid), cross-encoder se ~5 tak rerank karo LLM ke liye.

Kya reranking candidate set se missing relevant doc recover kar sakta hai?

Nahi — reranking sirf reorder karta hai; recall retrieval time par handle karna padta hai.

8. Connections

Vector Databases and ANN Search — dense candidate list supply karta hai.
BM25 and TF-IDF — sparse retriever ki scoring foundation.
Embeddings and Bi-encoders — dense vectors kaise produce hote hain.
Cross-encoders and Sentence Transformers — reranking model.
Retrieval-Augmented Generation (RAG) — woh pipeline jisme yeh feed hota hai.
Lost in the Middle - LLM context limits — kyun chota top-k matter karta hai.
Evaluation - Recall@k and MRR — retrieval quality kaise measure karte hain.