Chunking strategies for retrieval: Worked examples

Before we start, one promise: every symbol we use is one the parent already earned. Quick refresher so you never have to scroll:

Recall The six symbols we will reuse (peek if unsure)

$N$ ::: total number of tokens in the document (a token ≈ 4 English characters; see Tokenization). $c$ ::: chunk size — how many tokens go into each piece. $o$ ::: overlap — how many tokens two neighbouring chunks share. stride ::: the number of new tokens each chunk adds beyond its predecessor, equal to $c - o$ . $M$ ::: number of chunks produced. $s$ ::: the length (in tokens) of a fact/answer span we want to keep whole.

One more piece of notation appears constantly below, so let's earn it now:

And the two formulas everything below tests:

Definition The domain: legal ranges for the inputs

The formulas above only make sense inside these bounds — memorise them, because every "weird result" below traces back to leaving this box:

$c$ is a positive integer ( $c \geq 1$ tokens). A chunk is a whole number of tokens and must contain at least one; a zero or fractional chunk size is meaningless.
$0 \leq o < c$ . Overlap is a non-negative integer and must be strictly less than the chunk size. If $o \geq c$ the stride $c - o$ is zero or negative → the denominator breaks and $M$ is undefined/negative. You literally cannot advance the window if every chunk re-reads the whole previous one.
$N > o$ (and normally $N \geq 1$ ). If $N \leq o$ the numerator goes non-positive and $M \leq 0$ — nonsense; special-case empty/tiny input to $M \in {0, 1}$ .
For the overlap formula, $s$ is a positive integer ( $s \geq 1$ ), so that $s - 1$ is a sensible non-negative overlap. A fact of length 0 tokens is not a fact.
To contain a span of length $s$ you additionally need $c \geq s$ (a chunk must be at least as big as the fact). We prove why in Cell G.

The scenario matrix

Every chunk-sizing question is one of these cells. The examples below are labelled with the cell they hit, and together they touch all of them.

#	Cell (case class)	What makes it special	Example
A	Clean division, no overlap	$N$ is a multiple of $c$ , $o = 0$	Ex 1
B	Ugly remainder, no overlap	last chunk is partial → ceiling bites	Ex 2
C	Overlap present, non-integer division	the parent's headline case	Ex 3
D	Degenerate: document smaller than one chunk	$N \leq c$ → exactly 1 chunk	Ex 4
E	Limiting: overlap approaches chunk size	stride $\to 1$ , $M$ explodes; $o \geq c$ illegal	Ex 5
F	Zero / boundary: $o = 0$ vs the break probability	fact gets split	Ex 6
G	Real-world word problem: protect a fact (incl. $s > c$ )	choose $o$ from a span $s$	Ex 7
H	Exam twist: solve backwards for $c$ given a budget	inverse of the formula	Ex 8
I	Small-to-big: two different units	index unit ≠ return unit	Ex 9

The worked examples

Cell A — clean division, no overlap

Cell B — ugly remainder, no overlap

Cell C — overlap, non-integer division (the parent's headline)

Worked example Example 3 — reproducing the parent exactly

$N = 10, 000$ , $c = 500$ , $o = 50$ .

Forecast: overlap makes chunks advance slower than 500 each. So expect more than the $10000/500 = 20$ you'd get with no overlap. More than 20?

Stride $= c - o = 500 - 50 = 450$ . Why this step? Each chunk after the first re-reads 50 tokens from its neighbour, so it only adds $450$ new tokens. (Legal: $50 < 500$ .)
$M = ⌈ \frac{N - o}{c - o} ⌉ = ⌈ \frac{10000 - 50}{450} ⌉ = ⌈ \frac{9950}{450} ⌉$ . Why subtract $o$ on top? The very first chunk eats a full $c = 500$ tokens "for free"; after that, every chunk advances by only the stride. Subtracting the overlap once accounts for that head start. (Algebra: $c + (M - 1) (c - o) \geq N \Rightarrow M \geq 1 + \frac{N - c}{c - o} = \frac{N - o}{c - o}$ .)
$\frac{9950}{450} = 22.11 \dots \Rightarrow M = ⌈ 22.11 ⌉ = 23$ . Why ceiling again? Same reason as Cell B — the last partial chunk still needs a vector.

Verify: 23 chunks, stride 450, cover from token 0 up to at least $500 + 22 \times 450 = 10400 \geq 10000$ . ✔ And note $23 > 20$ , matching our forecast that overlap inflates the count. Overlap buys safety at the cost of a bigger index — a direct hit to Vector Databases and ANN Search storage and speed.

Cell D — degenerate: document smaller than one chunk

Worked example Example 4 — when the whole document fits in one chunk

$N = 300$ , $c = 500$ , $o = 50$ . How many chunks?

Forecast: the document (300 tokens) is smaller than one chunk (500). Can you have less than one chunk? Guess $M$ .

Numerator: $N - o = 300 - 50 = 250$ . Denominator: $c - o = 450$ . Why this step? We plug in blindly first, then sanity-check — the formula must not lie on small inputs. (Legal: $N = 300 > o = 50$ , and $o < c$ .)
$\frac{250}{450} = 0.555 \dots \Rightarrow M = ⌈ 0.555 ⌉ = 1$ . Why ceiling saves us here: $0.55$ rounds up to $1$ . A document must always produce at least one chunk — you can't index "half a chunk". The ceiling guarantees $M \geq 1$ whenever $N > o$ .

Verify: one chunk holds all 300 tokens (well under the 500 limit); no boundary is ever crossed, so overlap does nothing. $M = 1$ . ✔

Edge caution: if literally $N = 0$ (empty document), the formula gives $⌈(0 - o) / (c - o)⌉ = ⌈ negative ⌉ \leq 0$ — nonsense, and it violates the domain rule $N > o$ . In practice you special-case empty input to $M = 0$ .

Cell E — limiting behaviour: overlap approaches chunk size

Worked example Example 5 — pushing overlap toward the cliff

$N = 1000$ , $c = 500$ . Compute $M$ for $o = 400$ , then $o = 490$ , then imagine $o \to 499$ — and finally ask what happens at $o = 500$ .

Forecast: as overlap grows, stride shrinks. What happens to $M$ as stride heads toward 1? And what breaks if stride hits 0?

$o = 400$ : stride $= 100$ . $M = ⌈ \frac{1000 - 400}{100} ⌉ = ⌈ 6.0 ⌉ = 6$ . Why this step? Small stride means chunks barely creep forward — many chunks to cover the same 1000 tokens.
$o = 490$ : stride $= 10$ . $M = ⌈ \frac{1000 - 490}{10} ⌉ = ⌈ 51.0 ⌉ = 51$ . Why this step? Halving the stride roughly doubles the count; a tiny stride makes $M$ blow up.
Limit: as $o \to c - 1 = 499$ , stride $\to 1$ , and $M \to ⌈ \frac{N - o}{1} ⌉ \approx N - o$ . The chunk count approaches the token count — you are re-indexing nearly the whole document at every single token. Why this matters: $M \propto \frac{1}{c - o}$ . As $c - o \to 0$ , $M \to \infty$ .
Illegal boundary: at $o = c = 500$ the stride is exactly $0$ — the window never advances, so no finite number of chunks ever covers the document. The formula divides by zero. This is precisely why the domain rule demands $o < c$ ; overlap equal to (or bigger than) the chunk size is not a "very safe" setting, it is a broken setting.

Verify: see the figure below — the $M$ -vs- $o$ curve rises gently, then shoots vertically upward as $o$ nears $c = 500$ , and the dashed white wall at $o = 500$ marks the forbidden region where stride $\leq 0$ . ✔ Practical rule from the parent: keep $o \approx 10 - 20%$ of $c$ , far from this cliff.

Figure — Chunking strategies for retrieval

Cell F — the boundary that splits a fact ( $o = 0$ )

Worked example Example 6 — probability a fact gets cut

Chunk size $c = 200$ , overlap $o = 0$ . A fact spans $s = 14$ tokens and lands at a random position. What fraction of positions break it?

Forecast: boundaries sit every 200 tokens. A 14-token fact is short. High or low chance of a break? Guess a percentage.

Break probability (from the parent): $P (break) \approx \frac{s - 1}{c} = \frac{14 - 1}{200} = \frac{13}{200}$ . Why $s - 1$ and not $s$ ? A length- $s$ span occupies $s$ tokens but has $s - 1$ internal gaps between consecutive tokens. A boundary breaks the fact only if it falls into one of those internal gaps; there are $13$ of them.
$\frac{13}{200} = 0.065 = 6.5%$ . Why this matters: with $o = 0$ , about 1 in 15 such facts is silently split across two chunks and neither chunk retrieves it whole — a hidden recall failure in your Retrieval-Augmented Generation (RAG) system.

Verify: the picture below counts the gaps explicitly — 13 amber ticks, any of which a boundary can land in. ✔ To drive $P (break)$ to zero, jump to Cell G.

Cell G — real-world word problem: protect a fact (and the $s > c$ trap)

Worked example Example 7 — "this sentence must never be split"

A safety document has the critical line "The reactor melts down at 1,132°C, per standard §4.2." — about $s = 14$ tokens. Chunks are $c = 500$ . Choose an overlap $o$ that guarantees this line survives intact somewhere, then recompute $M$ for $N = 8000$ .

Forecast: from Cell F you know zero overlap risks a break. How much overlap exactly kills the risk?

First a feasibility check: is $c \geq s$ ? Here $500 \geq 14$ ✔. Why this step first? A chunk can only contain a fact that fits inside it. If the fact were longer than the chunk ( $s > c$ ), no overlap could ever hold it whole — a length- $s$ window simply doesn't fit in a length- $c$ box. So $c \geq s$ is a precondition before $o \geq s - 1$ even means anything. (We stress-test this in the note below.)
Minimum overlap: $o \geq s - 1 = 14 - 1 = 13$ . Why this step? Two neighbouring chunks share the last/first $o$ tokens. Any window of length $\leq o + 1$ that straddles their boundary lies entirely inside at least one of them. Setting $o + 1 \geq s$ gives $o \geq s - 1$ . So $o = 13$ is the exact threshold; the span can no longer be severed.
Practical choice: round up to a round number, $o = 20$ (a comfortable $4%$ of $c$ , still far from the Cell-E cliff, and legally $20 < 500$ ). Why round up? Real tokenizers wobble a token or two; a small safety margin costs almost nothing.
Recompute count with $N = 8000$ , $c = 500$ , $o = 20$ : stride $= c - o = 500 - 20 = 480$ , so $M = ⌈ \frac{N - o}{c - o} ⌉ = ⌈ \frac{8000 - 20}{480} ⌉ = ⌈ \frac{7980}{480} ⌉ = ⌈ 16.625 ⌉ = 17.$ Why this step? We confirm the safety choice doesn't blow up the index — $⌈ 16.625 ⌉ = 17$ chunks is barely more than the $16$ we'd get with $o = 0$ ( $⌈ 8000/500 ⌉ = 16$ ). The fact-protection is essentially free.

Verify: $500 + 16 \times 480 = 8180 \geq 8000$ ✔ (17 chunks cover the whole document). And $P (break) \approx (s - 1) / c$ is now irrelevant because the overlap structurally contains any 14-token window. ✔

Cell H — exam twist: solve backwards for $c$

Worked example Example 8 — given a chunk budget, find the size

Your vector DB budget allows at most $M = 40$ chunks for a document of $N = 12, 000$ tokens. You want overlap $o = 50$ . Find the smallest $c$ that keeps $M \leq 40$ (smaller $c$ means more chunks, so there is a floor below which you bust the budget).

Forecast: smaller $c$ → more chunks. So there's a smallest $c$ below which $M$ exceeds 40. Solve for it.

Start from $M = ⌈ \frac{N - o}{c - o} ⌉ \leq 40$ . Drop the ceiling for the bound: $\frac{N - o}{c - o} \leq 40$ . Why drop the ceiling? Since $40$ is a whole number, $⌈ x ⌉ \leq 40 ⟺ x \leq 40$ (rounding up can't exceed 40 only if the raw value already doesn't). So the continuous inequality is exact here. (Legal: we will need $c > o = 50$ for the stride to be positive.)
Rearrange: $N - o \leq 40 (c - o) \Rightarrow c - o \geq \frac{N - o}{40} \Rightarrow c \geq o + \frac{N - o}{40}$ . Why this direction? We isolate $c$ ; a bigger required stride means a bigger chunk.
Plug in: $c \geq 50 + \frac{12000 - 50}{40} = 50 + \frac{11950}{40} = 50 + 298.75 = 348.75$ . Why round up? $c$ must be a whole number of tokens (domain rule) and must satisfy $\geq$ , so $c = 349$ is the smallest legal chunk size (and $349 > o = 50$ , so the stride stays positive).

Verify (plug back): with $c = 349$ , $o = 50$ : stride $= 299$ , $M = ⌈ \frac{12000 - 50}{299} ⌉ = ⌈ \frac{11950}{299} ⌉ = ⌈ 39.97 ⌉ = 40 \leq 40$ ✔. And one token smaller, $c = 348$ : stride $298$ , $⌈ 11950/298 ⌉ = ⌈ 40.1 ⌉ = 41 > 40$ ✗ — confirming 349 is the exact threshold.

Cell I — small-to-big: two different units

Worked example Example 9 — index unit ≠ return unit

Query: "What voltage does the pump need?" The document has a spec line "Pump P-7: 24 V DC" ( $s_{index} = 6$ tokens) inside a 400-token "Pump P-7 specifications" section. Using parent-document retrieval, which unit do we embed, and which do we hand the LLM?

Forecast: should the matching unit and the reading unit be the same size? Why or why not?

Index unit = the 6-token spec line. Embed this. Why this step? A tiny, single-idea chunk produces a sharp embedding vector (see Text Embeddings and Cosine Similarity) that aligns tightly with the query "what voltage" — high precision in nearest-neighbour search.
Return unit = the whole 400-token section, fetched via a parent pointer stored in metadata. Why this step? The LLM needs context to answer follow-ups ("...and at what current?") without a second retrieval. The 400 tokens comfortably fit the context window.
Decoupling check: retrieval precision is set by the small unit ( $s_{index} = 6$ ), while answer completeness is set by the large unit ( $400$ ). Two knobs, tuned independently. Why this step? This is the whole point of small-to-big: fixed-size chunking forces one size to serve both matching and generation badly; decoupling lets each job pick its own ideal size. That is exactly why the parent's FRS-DP mnemonic ranks Parent-document last — most sophisticated.

Verify (units sanity): matching operates on a 6-token embedding (precise); generation reads 400 tokens ( $400 ≪$ a typical $8000$ -token context) → both constraints satisfied, no second query needed. ✔ Conclusion: embed the small line, return the big section — you get the precision of a tiny chunk and the completeness of a large one at the same time, which no single fixed size can deliver.

Putting the matrix together

Active-recall flashcards

What does the ceiling operator

⌈ x ⌉

mean?

Round

x

up to the nearest whole number; it never rounds down, so any fractional leftover bumps the result up by one.

With

N = 2200

c = 500

o = 0

, how many chunks and why?

⌈ 2200/500 ⌉ = ⌈ 4.4 ⌉ = 5

; the ceiling keeps the 200-token tail as its own chunk.

Why does the chunk-count formula subtract

o

in the numerator?

The first chunk consumes a full

c

tokens; every later chunk adds only the stride

c - o

, so we credit that one-time head start by subtracting the overlap once.

What is the legal range of the overlap

o

0 \leq o < c

with

o

an integer; if

o \geq c

the stride

c - o

is zero or negative and

M

is undefined.

For a document smaller than one chunk (

N \leq c

), what is

M

Exactly 1 — the ceiling forces

M \geq 1

whenever

N > o

As overlap

o \to c

, what happens to the chunk count?

The stride

c - o \to 0

M \propto 1/ (c - o) \to \infty

; at

o = c

the window never advances and

M

is undefined.

Overlap needed to protect a 14-token fact (and the precondition)?

First need

c \geq s

; then

o \geq s - 1 = 13

so any window up to length

o + 1

straddling a boundary sits wholly inside one chunk.

Can any overlap protect a fact with

s > c

No — a length-

s

window cannot fit inside a length-

c

chunk; you must enlarge

c

to at least

s

first.

Break probability of a length-

s

span with no overlap?

P (break) \approx (s - 1) / c

, because the span has

s - 1

internal gaps a boundary can land in.

Ex 7: with

N = 8000

c = 500

o = 20

, how many chunks?

stride

= 480

M = ⌈ 7980/480 ⌉ = ⌈ 16.625 ⌉ = 17

Smallest chunk size

c

giving

M \leq 40

for

N = 12000

o = 50

c \geq o + (N - o) /40 = 50 + 298.75 = 348.75 \Rightarrow c = 349

In small-to-big retrieval, what are the two different units?

A tiny index unit for precise matching and a larger parent unit returned to the LLM for context.