Reranking and hybrid search

1. WHY do we need this at all?

A naive RAG pipeline does: embed query → nearest neighbours in vector DB → feed top-k to LLM. Two failure modes:

1. Dense (vector) search misses exact terms. Embeddings capture meaning, so a query for the error code "ERR_4092" or the drug "Metformin" may retrieve semantically "nearby" but literally wrong chunks. It has no notion of an exact keyword match.
2. Sparse (keyword) search misses meaning. A search for "car" will not match a doc that only says "automobile".

The 80/20 insight: retrieval quality — not prompt tweaking — is usually the single biggest lever on RAG accuracy. Hybrid + rerank often beats any prompt engineering.

2. The two retrievers — derived from first principles

2.1 Sparse: BM25 (WHAT and WHY)

We want to score how relevant a document $D$ is to a query with terms $q_1,\dots,q_n$. Intuition to build in:

• A term that appears more often in $D$ should raise the score (term frequency, TF) — but with diminishing returns (the 50th "cat" tells us little beyond the 5th).
• A term that is rare across the whole corpus is more discriminating (inverse document frequency, IDF). "The" is useless; "photosynthesis" is gold.
• Long documents get a term-count advantage unfairly, so we normalise by length.

Putting these three ideas into a formula:

Why this shape?

• The fraction $\frac{f(k_1+1)}{f+k_1(\dots)}$ saturates: as $f\to\infty$ the score $\to (k_1+1)\cdot\text{IDF}$, giving diminishing returns. That's the "50th cat" effect.
• $k_1$ controls how fast it saturates; $b$ controls how strongly length normalisation kicks in ($b=0$ ⇒ ignore length, $b=1$ ⇒ full normalisation).
• $\text{IDF}(q_i)=\ln\!\frac{N-n(q_i)+0.5}{n(q_i)+0.5}+1$, with $N$ = total docs, $n(q_i)$ = docs containing $q_i$ — rare terms get a big multiplier.

2.2 Dense: cosine similarity of embeddings

Embed query and doc into vectors $\mathbf{q},\mathbf{d}\in\mathbb{R}^m$ with a bi-encoder (each encoded independently), score by: $\text{sim}(\mathbf{q},\mathbf{d})=\frac{\mathbf{q}\cdot\mathbf{d}}{\|\mathbf{q}\|\,\|\mathbf{d}\|}$ Because documents are embedded offline once, this is fast (approximate nearest neighbour search) — but the query and doc never "see" each other, so subtle interactions are lost.

3. Merging the two lists — Reciprocal Rank Fusion (RRF)

The scores from BM25 and cosine are on incomparable scales. We can't just add them. RRF sidesteps this by using only the rank of each document in each list.

Why $1/(k+\text{rank})$? A document ranked #1 contributes $\frac{1}{k+1}$; ranked #100 contributes $\frac{1}{k+100}\approx 0$. So top positions dominate, and a doc that appears in both lists (even at middling rank) beats a doc that appears in only one. The constant $k$ softens the difference between rank 1 and rank 2 so it isn't winner-take-all.

4. Reranking with a cross-encoder (the precision step)

The winning architecture: retrieve broadly and cheaply (bi-encoder + BM25 → top-100), then rerank precisely and expensively (cross-encoder → top-5). You pay the expensive cost only on 100 items, not the whole corpus.

$\text{final top-}k = \text{TopK}_k\Big(\big\{\,\text{CrossEncoder}(q,d)\ :\ d\in\text{Hybrid-top-}100\,\big\}\Big)$

5. Common mistakes (steel-manned)

6. Feynman

7. Flashcards

8. Connections

• Vector Databases and ANN Search — supplies the dense candidate list.
• BM25 and TF-IDF — the sparse retriever's scoring foundation.
• Embeddings and Bi-encoders — how dense vectors are produced.
• Cross-encoders and Sentence Transformers — the reranking model.
• Retrieval-Augmented Generation (RAG) — the pipeline this feeds.
• Lost in the Middle - LLM context limits — why small top-k matters.
• Evaluation - Recall@k and MRR — how to measure retrieval quality.

Concept Map

Hinglish (regional understanding)