4.4.13 · D5Alignment, Prompting & RAG
Question bank — Chunking strategies for retrieval
Prerequisites worth having open: Retrieval-Augmented Generation (RAG), Text Embeddings and Cosine Similarity, Vector Databases and ANN Search, Context Window Limits of LLMs, Tokenization, Metadata Filtering in Retrieval.
True or false — justify
A big chunk always retrieves at least as well as a small one because it contains more information
False. One chunk becomes exactly one embedding vector; a big chunk averages many topics into a blurry vector that matches any single query weakly, so precision drops.
Setting overlap to zero is safe as long as chunks are large
False. Size does not fix straddling — any fact whose tokens cross a boundary is split no matter how big the chunk; only overlap covers the seam.
Two consecutive chunks with overlap physically store the shared tokens twice
True. Overlap is literal duplication in the index; that is the storage cost you pay to protect boundary-straddling spans.
Chunking on 500 characters is equivalent to chunking on 500 tokens
False. Embedding models and context windows count tokens, not characters (~4 chars/token in English, wildly different for code or other languages), so the two give different chunk sizes.
Semantic chunking is strictly better than fixed-size chunking
False. It follows meaning but costs one embedding call per sentence at index time; for uniform prose or logs the fixed-size method is cheaper and nearly as good.
Recursive/structure-aware chunking can still fall back to a hard mid-sentence cut
True. It prefers natural separators (paragraph, then sentence, then word) but must hard-cut any piece that still exceeds the size limit .
In small-to-big retrieval, the unit you embed and the unit you send to the LLM are the same
False. You embed the tiny precise chunk for a sharp nearest-neighbour match but return its larger parent for context — the two units are deliberately decoupled.
Increasing overlap always increases the number of chunks
True. Larger overlap shrinks the stride , and grows as the stride shrinks, so more chunks (and more index storage) result.
Chunking quality sets a ceiling that later pipeline stages cannot exceed
True. Retrieval can only return the units you indexed; if the right idea is never a coherent chunk, no re-ranker or LLM downstream can recover it.
Spot the error
"I set overlap and chunk size , so each chunk is well protected."
The stride is , meaning every chunk starts at the same place — the loop never advances and is undefined. Overlap must be strictly less than .
"My answer spans tokens, so I need overlap ."
Off by one — the guarantee is , because the overlap only needs to cover the straddle, i.e. a window of length needs .
"I embedded whole 50-page PDFs to keep everything in context."
The retrieval unit is now a whole document, so a query returns 50 pages of noise around 2 useful sentences, and the single vector is far too blurry to match sharply.
"I chunked one-word-per-unit for maximum precision."
Each unit loses all surrounding context ("it was invented in 1876" — what was?), so the embedding is ambiguous and the LLM cannot reason from the fragment.
"Section headings are just formatting, so I stripped them before chunking."
The heading path is valuable metadata for document-structure chunking and for Metadata Filtering in Retrieval; discarding it throws away the very signal that keeps each chunk a coherent, filterable idea.
"I chunk with my own character splitter but embed with the model's tokenizer — same thing."
The chunk sizes you plan in characters won't match the token counts the model and context window actually see; you must chunk on the same tokenizer the model uses.
"Semantic chunking uses cosine similarity over a threshold to start a new chunk."
It starts a new chunk when cosine distance between consecutive sentences exceeds a threshold (a topic shift) — high similarity means keep them together, not split.
Why questions
Why does the chunk-count formula subtract the overlap once in the numerator, ?
The very first chunk consumes a full tokens; every later chunk advances only by the stride , so solving leaves that single correction in the numerator.
Why is and not ?
A span of tokens has internal gaps where a boundary could fall and split it; a boundary at either outer edge leaves the span intact, so only of the positions are dangerous.
Why does a smaller chunk size mean slower nearest-neighbour search?
Smaller chunks produce more vectors (), enlarging the index the ANN index must traverse, which raises both storage and query cost.
Why does small-to-big retrieval give "the best of both worlds"?
The tiny indexed chunk yields a sharp embedding that nails the cosine match, while the returned parent supplies enough context for the LLM to answer follow-ups without a second query.
Why do we care about token counts specifically rather than word counts?
Both the embedding model's input limit and the context window are measured in tokens by the tokenizer, so token budget is what actually constrains what fits.
Why can respecting natural boundaries improve embeddings even when it makes chunks uneven in size?
Boundaries aligned with paragraphs or sentences keep each chunk one coherent idea, so its vector represents a single meaning sharply — worth more than uniform token counts.
Edge cases
What happens when (overlap equals chunk size)?
The stride is zero, so consecutive chunks never advance and the formula divides by zero — a degenerate configuration you must forbid ().
What if the whole document is shorter than one chunk, ?
You get exactly one chunk containing the entire document; the ceiling formula returns and overlap is irrelevant since there is no boundary.
What if the answer span is longer than the chunk size ?
No single chunk can hold it intact regardless of overlap; you must either raise above or switch to small-to-big so the parent unit carries the full span.
What happens to the last chunk when is not a multiple of the stride?
It is partial but still gets its own vector — this is exactly why the count uses a ceiling , rounding a fractional last chunk up to one whole chunk.
What if two adjacent sentences are on the same topic in semantic chunking?
Their cosine distance stays below the threshold, so no new chunk starts and they are kept together — the method only splits at genuine topic shifts.
What is the effect of setting overlap larger than any real answer span, say for ?
Every relevant span is safely intact, but you have wasted storage and slowed search by duplicating far more tokens than needed — overlap around – of is the practical sweet spot.
What happens if a query's answer requires two facts stored in two different chunks?
A single top-1 retrieval misses one fact; you rely on top- retrieval to pull both chunks, or on small-to-big to return a shared parent covering both.
Recall One-line summary of the traps
Bigger is not safer (blurry vectors), zero overlap splits facts, characters ≠ tokens, similarity ≠ distance, and every degenerate case (, , ) has a specific right answer — none is "just fine".