Noesis
4.4.13Alignment, Prompting & RAG

Chunking strategies for retrieval

1,950 words9 min readdifficulty · medium2 backlinks
🇮🇳 Yeh note Hinglish mein padho →

WHAT is chunking?

The RAG pipeline is: document → chunk → embed → store in vector DB → (query) embed → nearest-neighbour search → top-k chunks → LLM. Chunking sits at the very front — every downstream quality ceiling is set here.

WHY the number-of-tokens matters (a mini derivation)

Suppose a document has NN tokens and we split into chunks of size cc tokens. Number of chunks:

M=⌈Nc⌉M = \left\lceil \frac{N}{c} \right\rceil

Why ceiling? the last chunk may be partial but still needs its own vector.

Each chunk is embedded to one vector. So the index size grows as M∝1/cM \propto 1/c: smaller chunks ⇒ more vectors ⇒ more storage and slower search, but finer resolution.

Now the probability a needed fact survives in a single retrievable chunk. If the answer spans ss tokens and we place chunk boundaries without overlap, the answer is split whenever a boundary lands inside it. The fraction of "boundary positions" that break a length-ss span:

P(break)≈s−1cP(\text{break}) \approx \frac{s-1}{c}

Effective stride (new tokens per chunk) is c−oc-o, so the chunk count becomes:

M=⌈N−oc−o⌉M = \left\lceil \frac{N - o}{c - o} \right\rceil

Why this form? the first chunk consumes cc tokens; each later chunk advances by the stride c−oc-o. Solving c+(M−1)(c−o)≥Nc + (M-1)(c-o) \ge N gives the formula.

Figure — Chunking strategies for retrieval

HOW: the main strategies (in 80/20 order)

1. Fixed-size (token/char) chunking

Split every c−oc-o tokens with overlap oo. Simple, fast, predictable index size.

  • When: uniform prose, logs, transcripts.
  • Weakness: cuts mid-sentence → semantic blur.

2. Recursive / structure-aware chunking

Try to split on the biggest natural separator first (\n\n paragraphs), then sentences, then words, only falling back to hard cuts when a piece exceeds cc.

  • Why better: boundaries align with meaning, so each chunk is a coherent idea.
  • When: articles, docs, markdown.

3. Semantic chunking

Embed each sentence; start a new chunk when the cosine distance between consecutive sentences exceeds a threshold (a topic shift).

  • Why: boundaries follow meaning, not token counts.
  • Cost: many extra embedding calls.

4. Document-structure chunking

Use headings/sections/tables/code-blocks as units (one chunk per section). Attach the heading path as metadata.

  • When: technical docs, legal contracts, wikis.

5. Small-to-big / parent-document retrieval

Index small chunks (precise matching) but return the larger parent for context.

  • Why: decouples the retrieval unit from the generation unit — best of both.

Worked examples

Common mistakes (steel-manned)

Recall Feynman: explain to a 12-year-old

Imagine a huge library book with one amazing fact hidden inside. Instead of handing your friend the whole book, you tear it into index cards — each card holds one clear idea. Now when your friend asks a question, you can find the one right card fast. If the cards are too big they hold too many ideas and are confusing; too small and each card doesn't make sense on its own. And you let neighbouring cards share a sentence so a fact split across two cards is never lost. That's chunking.

Active-recall flashcards

What is chunking in a RAG pipeline?
Splitting documents into smaller pieces that are individually embedded and indexed, so retrieval returns relevant chunks, not whole documents.
Why does a chunk that is too large hurt retrieval?
Its single embedding averages many ideas into a blurry vector that matches queries weakly, lowering precision and wasting context tokens.
Why does a chunk that is too small hurt retrieval?
It loses surrounding context, so the embedding is ambiguous and the LLM lacks enough to reason.
Formula for number of chunks with overlap o and size c over N tokens?
M=⌈(N−o)/(c−o)⌉M=\lceil (N-o)/(c-o)\rceil
Minimum overlap to keep a span of s tokens intact across a boundary?
o≥s−1o \ge s-1
What is the effective stride between chunk starts?
c−oc - o (chunk size minus overlap).
Name the FRS-DP strategies.
Fixed-size, Recursive/structure-aware, Semantic, Document-structure, Parent-document (small-to-big).
How does semantic chunking decide boundaries?
Start a new chunk when cosine distance between consecutive sentence embeddings exceeds a threshold (topic shift).
What is small-to-big / parent-document retrieval?
Index small precise chunks for matching but return their larger parent chunk for context.
Why chunk on tokens not characters?
Embedding models and context windows count tokens (~4 chars/token in English, varies by language/code), so char limits mismatch the model.

Connections

  • Retrieval-Augmented Generation (RAG)
  • Text Embeddings and Cosine Similarity
  • Vector Databases and ANN Search
  • Context Window Limits of LLMs
  • Tokenization
  • Metadata Filtering in Retrieval

Concept Map

split by

produces

embedded to

stored in

nearest-neighbour

fed to

too big causes

too small causes

size c sets

boundaries split spans

fixed by overlap

gives stride

Source document

Chunking

Right-sized units

One vector per chunk

Vector DB

Top-k chunks

LLM answer

Semantic blurring

Lost context

Index size M ~ 1 over c

P break ~ s-1 over c

Overlap o >= s-1

Stride c minus o

Hinglish (regional understanding)

Intuition Hinglish mein samjho

Dekho, RAG mein sabse pehla aur sabse important step hai chunking — document ko chhote tukdon mein todna. Socho tumhare paas ek 50-page PDF hai. Agar tum poora document ek hi vector mein embed kar doge, to woh vector saari ideas ka average ban jaata hai — kisi bhi query se sharp match nahi karega. Aur agar tum ek-ek word ka chunk banaoge, to context hi khatam — "1876 mein invent hua" par kya invent hua, pata hi nahi chalega. Isliye chunk ka size ek tradeoff hai: ek chunk mein ek clear idea hona chahiye, roughly 200-500 tokens.

Ab overlap ka jugaad samjho. Bina overlap ke, agar koi important sentence do chunks ke beech ki boundary par aa gaya, to woh do tukdon mein tut jaayega aur retrieval usko pura kabhi nahi laayega. Formula simple hai: agar answer span ss tokens ka hai, to overlap o≥s−1o \ge s-1 rakho — tabhi guarantee hai ki woh span kisi ek chunk mein pura milega. Practically 10-20% overlap rakhna safe hai.

Strategies yaad rakhne ke liye mnemonic hai FRS-DP: Fixed-size (simple, tez), Recursive (paragraph/sentence pe todo — meaning preserve hoti hai), Semantic (topic change pe naya chunk), Document-structure (heading/section wise), aur Parent-document ya small-to-big (chhota chunk se match karo, bada parent return karo context ke liye). 80/20 rule: bas fixed-size-with-overlap aur recursive achhe se seekh lo — 80% cases yahi solve kar dete hain. Ek cheez kabhi mat bhoolna — chunk tokens par karo, characters par nahi, kyunki model tokens hi ginta hai.

Go deeper — visual, from zero

D1FoundationsEvery symbol from zero · 2,296 wordsD2Visual walkthroughThe derivation in pictures · 1,999 wordsD3Worked examplesEvery scenario covered · 4,197 wordsD4ExercisesGraded, full solutions · 2,548 wordsD5Question bankConcept traps, quick reveals · 1,417 words

Test yourself — Alignment, Prompting & RAG

L1 RecognitionL2 RecallL3 ProductionL4 ApplicationL5 Mastery

Connections

Retrieval-Augmented Generation (RAG) architectureAI-ML · 4.4.11Retrieval-Augmented Generation (RAG) architectureAI-ML · 4.4.11Vector databases and embeddingsAI-ML · 4.4.12