Exercises — Chunking strategies for retrieval

Answer: the chunk step. A retriever can only return the units you indexed, so if a unit is badly sized (too big = blurry vector, too small = no context) no downstream stage can recover the lost precision or context. Chunking sits at the very front, so its quality is an upper bound on everything after it.

(a) → chunk too big — one vector must average many ideas, giving semantic blurring. (b) → chunk too small — the chunk lost its surrounding context, so the embedding is ambiguous.

Fixed-size → Recursive / structure-aware → Semantic → Document-structure → Parent-document (small-to-big).

With no overlap the stride equals $c$, so use $M=\lceil N/c\rceil$. $M=\lceil \frac{8000}{400}\rceil =\lceil 20\rceil =20\text{ chunks.}$ It divides evenly, so no rounding was needed — but we still write the ceiling out of habit, because most real numbers won't divide cleanly.

Overlap changes the stride to $c-o=400-40=360$. Use the overlap formula: $M=\lceil \frac{N-o}{c-o}\rceil =\lceil \frac{8000-40}{360}\rceil =\lceil \frac{7960}{360}\rceil =\lceil 22.11\dots \rceil =23\text{ chunks.}$ Why more chunks than Q1? Each chunk now advances only 360 new tokens instead of 400, so you need more of them to cover the same document. Overlap costs storage — that's the price of protecting facts near boundaries.

Use $o\ge s-1$: $o\ge 14-1=13.$ So $o=13$ is the minimum; a safe practical choice is $o\approx 15$–$20$. Why $s-1$ and not $s$? Two neighbouring chunks share $o$ tokens; a window of length up to $o+1$ that straddles the boundary is fully inside at least one chunk. Setting $o+1\ge s$ gives $o\ge s-1$.

Use $P(\text{break})\approx \frac{s-1}{c}=\frac{14-1}{500}=\frac{13}{500}=0.026=2.6\%$. Small, but over thousands of documents that's many lost facts — which is exactly why overlap exists.

$M_{800}=\lceil 16000/800\rceil =20$. $M_{400}=\lceil 16000/400\rceil =40$. Factor $=40/20=2\times$. This matches $M\propto 1/c$: halve $c$, double the vectors. Cost: twice the storage and slower nearest-neighbour search in the vector DB. Benefit: finer resolution — each vector represents one tighter idea, improving precision.

The fact has span $s=4$, so it needs $o\ge s-1=3$.

• $o=1$: the shared band is only 1 token wide; the 4-token fact cannot fit entirely inside either chunk → split, not retrievable intact.
• $o=3$: the shared band is 3 tokens; any window of length $\le o+1=4$ straddling the boundary fits fully in one chunk → the fact survives in the right-hand chunk (or left, depending on position). Look at the green band in the figure widening from 1 to 3 — that widening is literally the guarantee.

Fixed-size cuts land at arbitrary token positions — mid-sentence, mid-table-row, or mid-code-block — producing chunks that are syntactically broken and semantically blurry. Recursive/structure-aware chunking splits on the biggest natural separator first (\n\n, then sentences), and document-structure chunking makes one chunk per section and attaches the heading path as metadata. Both keep each chunk a coherent idea, so embeddings are sharper and retrieval is more precise.

Benefit: boundaries follow meaning (a genuine topic shift), not an arbitrary token count, so chunks are maximally coherent. Cost: you must embed every sentence to compute those distances — many extra embedding API calls, so it is slower and more expensive to build the index. See Text Embeddings and Cosine Similarity for the distance measure.

• (a) Strategy: Parent-document / small-to-big, layered on document-structure chunking (clauses are natural units).
• (b) Index unit: a small precise piece — e.g. the single obligation sentence "Either party may terminate with 30 days' written notice." Small chunk → sharp embedding → nails the nearest-neighbour match.
• (c) Return unit: the whole parent clause §7.3 — enough context for the LLM to answer and handle follow-ups without a second query.
• (d) Metadata: the clause/section path (e.g. section = "7.3 Termination"), enabling metadata filtering to restrict search to relevant sections. Why this design: it decouples the retrieval unit (tiny, precise) from the generation unit (large, contextual) — the defining advantage of small-to-big.

(i) Protect the span: need $o\ge s-1=29$. (ii) $20\%$ cap: $0.20\times 512=102.4$, so any $o\le 102$ is allowed. A clean choice satisfying both is $o=30$ (or the tidy $o\approx 50$). Take $o=30$: stride $=512-30=482$. $M=\lceil \frac{N-o}{c-o}\rceil =\lceil \frac{50000-30}{482}\rceil =\lceil \frac{49970}{482}\rceil =\lceil 103.67\dots \rceil =104\text{ chunks.}$ Why these steps: the span constraint sets the floor on $o$; the $20\%$ rule sets the ceiling; picking anything in between and plugging into the stride formula gives the index size.

Let the chunks start at token positions $0,(c-o),2(c-o),\dots$. After $M$ chunks the last chunk ends at position: $c+(M-1)(c-o).$ (The first chunk contributes its full length $c$; every one of the remaining $M-1$ chunks advances the coverage by one stride $c-o$.) We need this to reach or pass the final token $N$: $c+(M-1)(c-o)\ge N.$ Isolate $M-1$: $(M-1)(c-o)\ge N-c\Rightarrow M-1\ge \frac{N-c}{c-o}$. Add $1$: $M\ge 1+\frac{N-c}{c-o}=\frac{(c-o)+(N-c)}{c-o}=\frac{N-o}{c-o}.$ Since $M$ must be a whole number and we need the smallest such $M$, take the ceiling: $M=\lceil \frac{N-o}{c-o}\rceil .■$

(a) $o=0$: $M=\lceil (N-0)/(c-0)\rceil =\lceil N/c\rceil$ — the no-overlap formula falls straight out. ✓ Consistent. (b) $N\le c$: the whole document fits in one chunk. Plug $N=c$: $M=\lceil (c-o)/(c-o)\rceil =\lceil 1\rceil =1$. For $N<c$ the numerator $N-o<c-o$, so the ratio is $<1$ and the ceiling is $1$. ✓ Exactly one chunk — correct. (c) $o\to c$: the stride $c-o\to 0^{+}$, so $M=\lceil (N-o)/(c-o)\rceil \to \infty$. Interpretation: chunks barely advance, so you need infinitely many to cover the document — the mathematical warning against runaway overlap.

True token length: $40,000\text{ chars}÷4\text{ chars/token}=10,000$ tokens. Teammate expected (treating $c=1000$ as characters): $\lceil 40000/1000\rceil =40$ chunks. Reality (model chunks by tokens, $c=1000$ tokens): $\lceil 10000/1000\rceil =10$ chunks. The mismatch is a $4\times$ error. Lesson: always chunk on the same tokenizer the model uses — see Tokenization — because "size" means tokens, and $\approx 4$ chars/token holds only for English (code and other languages differ wildly). This also interacts with Context Window Limits of LLMs: mis-sized chunks can silently overflow the window.

Need $M=\lceil N/c\rceil \le 120$. The binding condition is $N/c\le 120$ (ceiling only ever rounds up, so satisfying the un-ceiled inequality with a whole-token boundary is enough here). Solve for $c$: $c\ge \frac{N}{120}=\frac{60000}{120}=500.$ Smallest allowed chunk size: $c=500$ tokens. Check: $M=\lceil 60000/500\rceil =\lceil 120\rceil =120\le 120$. ✓ If you tried $c=499$, $M=\lceil 60000/499\rceil =\lceil 120.24\rceil =121>120$ — over budget. So $c=500$ is exactly on the edge.

Chunking sets the quality ceiling (L1). You size the index with $\lceil (N-o)/(c-o)\rceil$ and protect facts with $o\ge s-1$ (L2). Smaller chunks trade storage for precision, and structure beats blind cuts (L3). Real pipelines decouple a tiny precise index unit from a large contextual return unit (L4). And the formula behaves sensibly at every edge — no overlap, tiny docs, runaway overlap, token-vs-char, and hard budgets — as long as you never drop the ceiling (L5).