Vocabulary size tradeoffs: Worked examples — AI-ML deep dive

This page is a drill hall. The parent note built the two cost formulas — embedding table size and attention cost. Here we throw every kind of case at those formulas until none can surprise you.

Before we start, let us re-anchor the two tools so no symbol is used unearned.

Definition The two costs, in plain words

$V$ = vocabulary size = how many distinct tokens the model knows. A whole number.
$d$ = embedding dimension = the length of the vector that stands in for each token. A whole number.
$n$ = sequence length = how many tokens a given piece of text becomes after tokenizing.
$L$ = number of transformer layers = how many times the attention machinery is stacked and repeated. Each layer builds its own $n \times n$ score grid, so costs get multiplied by $L$ .
$\overset{ˉ}{ℓ}$ = average token length = the mean number of characters each token swallows. A byte token has $\overset{ˉ}{ℓ} = 1$ ; a subword like "learn" has $\overset{ˉ}{ℓ} = 5$ . Since a text of $C$ characters splits into $n = C / \overset{ˉ}{ℓ}$ tokens, bigger $\overset{ˉ}{ℓ}$ means fewer tokens.
Embedding table cost $= V \times d$ parameters. Picture a grid: $V$ rows (one per token), $d$ columns (one per vector slot). Total boxes = $V \times d$ .
Attention cost $\propto n^{2} d$ per layer. Picture an $n \times n$ grid of "how much does token $i$ look at token $j$ " scores — that grid has $n^{2}$ cells, and it must be built once per layer (so once per each of the $L$ layers).

The single most important sign-of-life fact: embedding cost grows with $V$ (a fixed one-time price), attention cost grows with $n^{2}$ (a per-batch, per-use price). Every example below is really asking "which side of this seesaw tips?"

The scenario matrix

Every problem this topic can ask lands in one of these cells. The worked examples afterward each carry a tag [Cell X] telling you which one it fills.

Cell	Case class	What makes it tricky
A	Plain embedding-table sizing	Just $V \times d$ — but watch units (M vs B params)
B	Plain attention memory sizing	The scalar-vs-vector trap (do NOT multiply by $d$ )
C	Breakeven: small vs large vocab	Compare permanent params against accumulated batch savings
D	Degenerate low end: byte-level, $V \approx 256$	Tiny table, huge $n$ — attention blows up
E	Degenerate high end: $V \to$ whole words	Short $n$ , giant table, rare-token starvation
F	Zero / limiting input	Empty text ( $n = 0$ ), one-token text, $V = 1$
G	Real-world word problem	Multilingual / domain, must choose a number
H	Exam twist	A hidden wrong assumption you must catch

We cover A through H across 9 examples: A (Ex 1), B (Ex 2), C (Ex 3), D (Ex 4), E (Ex 5), F (Ex 6), G (Ex 7 — with its geometric figure), the U-shaped seesaw synthesis (Ex 8), and H (Ex 9).

The examples

Worked example Example 1 — Table sizing done right

[Cell A] A BERT-base-style model has $V = 30000$ and $d = 768$ . How many parameters live in the embedding table, and what fraction of a 110M-parameter model is that?

Forecast: guess — is the embedding table a rounding error, or a big chunk? Write down your guess (single-digit percent? twenty-something percent?).

Multiply: $V \times d = 30000 \times 768 = 23, 040, 000$ parameters. Why this step? The table is literally a $V$ -by- $d$ grid of numbers; every box is one parameter. No layers, no batches — this is a static count.
Convert: $23, 040, 000 = 23.04 M$ . Why this step? Model sizes are quoted in millions; keeping raw digits invites arithmetic slips.
Fraction: $\frac{23.04}{110} \approx 0.209 = 20.9%$ . Why this step? The question asks "what fraction", so we compare to the total budget.

Verify: Sanity check — the embedding table is often the single largest weight matrix in small models, and $\approx 21%$ is entirely believable (people frequently tie input and output embeddings precisely because this slab is so big). Units: parameters ÷ parameters = dimensionless fraction. ✓

Worked example Example 2 — Attention memory, avoiding the scalar trap

[Cell B] Sequence length $n = 2000$ , batch size $8$ , $L = 12$ layers, scores stored in float16 (2 bytes each). How much memory do the raw attention score matrices take? Then compute the WRONG number a beginner gets by also multiplying by $d = 512$ .

Forecast: guess the right answer to the nearest hundred MB, and guess how many times too big the wrong answer is.

Score matrix per layer per batch-item: $n \times n = 200 0^{2} = 4, 000, 000$ scalars. Why this step? A score answers "how much does token $i$ attend to token $j$ " — one number per pair. That is an $n \times n$ grid of scalars, not vectors.
Multiply out all copies: $8 \times 12 \times 4, 000, 000 = 384, 000, 000$ scalars. Why this step? Each of 8 sequences, in each of 12 layers, needs its own score grid.
Bytes: $384, 000, 000 \times 2 = 768, 000, 000$ bytes $\approx 768 MB$ . Why this step? float16 = 2 bytes per number.
The wrong number: multiply by $d = 512$ → $768 MB \times 512 = 393, 216 MB \approx 384 GB$ . Why this step? This is the classic error — treating each score cell as a $d$ -vector. It inflates the answer by exactly $512 \times$ .

Verify: Ratio $\frac{384 GB}{768 MB} = \frac{393216}{768} = 512 = d$ . ✓ The blunder factor is exactly $d$ — a clean tell that you accidentally carried the embedding dimension into a scalar count.

Worked example Example 3 — Breakeven: does a bigger vocab pay off?

[Cell C] Small: $V_{s} = 32000$ , $n_{s} = 2000$ . Large: $V_{l} = 128000$ , $n_{l} = 1400$ (30% shorter sequences from domain merges). $d = 512$ , batch size $8$ , $L = 12$ , float16 (2 bytes per stored number). Over how many batches does the large vocab break even, if we count everything in bytes?

Forecast: guess — tens of batches? thousands? never?

Extra permanent params in bytes: $(V_{l} - V_{s}) \times d = (128000 - 32000) \times 512 = 49, 152, 000$ params; at 2 bytes each that is $49, 152, 000 \times 2 = 98, 304, 000$ bytes $\approx 98.3 MB$ . Why this step? This is the one-time price the large vocab pays forever. To compare it to activation memory honestly we must express it in the same unit — bytes — not raw parameter counts. float16 params also cost 2 bytes each.
Attention score bytes, small: $8 \times 12 \times 200 0^{2} \times 2 = 768, 000, 000$ bytes. Large: $8 \times 12 \times 140 0^{2} \times 2 = 376, 320, 000$ bytes. Why this step? We size the recurring activation footprint per batch, again in bytes, so both sides of the comparison share one currency.
Savings per batch: $768, 000, 000 - 376, 320, 000 = 391, 680, 000$ bytes $\approx 391.7 MB$ per batch. Why this step? Shorter sequences shrink the $n^{2}$ grid; the byte difference is what the big vocab buys back each batch.
Breakeven batch count $= \frac{extra param bytes}{saving bytes per batch} = \frac{98 , 304 , 000}{391 , 680 , 000} \approx 0.251$ batches. Why this step? We equate the fixed byte cost to accumulated byte savings. Because both numerator and denominator are now genuine byte counts, the ratio is a valid, unit-consistent comparison — not a hand-wave about "stored numbers".

Verify: Both scaled by the same 2 bytes/number, so the byte ratio equals the raw-count ratio: $\frac{49.152 M}{195.84 M} = \frac{98.304 M bytes}{391.68 M bytes} \approx 0.251$ . ✓ Breakeven under one batch means the large vocab's extra table is repaid almost immediately by shorter-sequence activation savings. Caveat: this compares storage footprint only — it ignores that params are reused across all batches while activations are transient, so treat $\approx 0.25$ batches as a memory-pressure estimate, not a wall-clock claim. (Answer $\approx 0.25$ batches.)

Worked example Example 4 — Byte-level degenerate low end

[Cell D] A 1000-character English sentence tokenized at the byte level: $V = 256$ , so roughly $n = 1000$ tokens. Compare its attention-score count to the same text at $V = 32000$ BPE where $\overset{ˉ}{ℓ} \approx 4$ chars/token. $L = 1$ , batch 1 for cleanliness.

Forecast: guess the ratio of attention work between byte-level and BPE. 4×? 16×?

Byte-level $n_{byte} \approx 1000$ (one token per character). Why this step? $V = 256$ can only cover single bytes; nothing merges, so token count = char count.
BPE $n_{bpe} \approx \frac{1000}{4} = 250$ . Why this step? Average token length $\overset{ˉ}{ℓ} = 4$ means each token eats 4 characters, so $n = chars / \overset{ˉ}{ℓ}$ .
Attention-score ratio $= \frac{n _{byte}^{2}}{n _{bpe}^{2}} = (\frac{1000}{250})^{2} = 4^{2} = 16$ . Why this step? Attention scales as $n^{2}$ ; a $4 \times$ longer sequence costs $16 \times$ the score grid.
Embedding table ratio (the other side): $\frac{256}{32000} = 0.008$ . Why this step? Byte-level's table is only $0.8%$ the size — the degenerate low end saves table memory but pays $16 \times$ in attention.

Verify: The $n^{2}$ law says linear length change squares in cost: $4^{2} = 16$ . ✓ This is exactly the "waste compute on long sequences" warning from the parent note, now quantified.

Worked example Example 5 — Whole-word degenerate high end

[Cell E] Push vocab up until common words are single tokens: $V = 250000$ , $d = 12288$ (GPT-3-scale width). (a) Embedding table size. (b) Suppose 40% of these tokens each appear fewer than 5 times in a training corpus — how many parameters are "starved"?

Forecast: guess whether the table is under 1B or over 1B params, and how many params are starved.

Table: $V \times d = 250000 \times 12288 = 3, 072, 000, 000 \approx 3.07 B$ parameters. Why this step? Same grid rule; at large $d$ the table alone rivals a whole small model.
Starved rows: $0.40 \times 250000 = 100000$ tokens. Why this step? A rare token's row is updated only a handful of times, so its 12288 numbers stay near-random.
Starved params: $100000 \times 12288 = 1, 228, 800, 000 \approx 1.23 B$ parameters barely trained. Why this step? Multiply starved rows by row width to see the wasted capacity.

Verify: $1.23 B$ of $3.07 B$ is $\frac{1.2288}{3.072} = 0.40 = 40%$ — matches the starved fraction exactly, as it must (same $d$ per row). ✓ This is the concrete face of "rare tokens have poor embeddings" from the parent's mistake box.

Worked example Example 6 — Zero and limiting inputs

[Cell F] Handle the edge cases the formulas must not choke on. (a) Empty text. (b) A single-token text. (c) A degenerate vocabulary of $V = 1$ .

Forecast: guess which of these gives zero cost and which gives a useless but finite model.

Empty text: $n = 0$ . Attention scores $= n^{2} = 0$ . Embedding table still $V \times d$ (it exists regardless of any text). Why this step? Sequence cost depends on the text; table cost depends on the vocab. They are independent — the empty case cleanly separates them.
Single token: $n = 1$ . Attention grid $= 1^{2} = 1$ score (a token attending to itself). Cost is minimal but nonzero. Why this step? Even one token forms a $1 \times 1$ attention grid — the formula degrades gracefully to a single self-score.
$V = 1$ : table $= 1 \times d = d$ . Every piece of text becomes a repeat of the same lone token; the model can encode nothing. Why this step? $V = 1$ is the information-density floor: zero bits per token, so $n$ can be huge yet carry no meaning.

Verify: All three stay finite: $0$ , $1$ , and $d$ respectively. No division by $V$ or by $n$ occurs anywhere in the cost formulas, so no case explodes. ✓

Worked example Example 7 — Multilingual word problem

[Cell G] A 100-character Chinese text. Character-level: $V \approx 1000$ , one token per character so $n = 100$ . Subword BPE: $V = 50000$ , giving $n \approx 35$ . $d = 768$ . Which wins on total memory pressure, and by how much on attention?

Forecast: guess — does the tiny character vocab win (small table) or the subword vocab (short sequences)?

Attention-score ratio: $\frac{n _{char}^{2}}{n _{sub}^{2}} = \frac{10 0 ^{2}}{3 5 ^{2}} = \frac{10000}{1225} \approx 8.16$ . Why this step? Attention scales as $n^{2}$ . Why can we drop $L$ and the batch size here? Both tokenizations run through the same model — same layer count $L$ , same batch size $b$ — so each side's full attention memory is $b L n^{2}$ . Forming the ratio, the common factor $b L$ cancels top and bottom, leaving pure $\frac{n _{char}^{2}}{n _{sub}^{2}}$ . We drop them because they are identical on both sides, not because they are zero.
Subword table size: $50000 \times 768 = 38, 400, 000 = 38.4 M$ params. Character table size: $1000 \times 768 = 768, 000 = 0.768 M$ params. Why this step? We size both tables explicitly with the same grid rule before comparing them. Here $d = 768$ is also common to both, so it too cancels in any table ratio — but we keep it to report absolute sizes.
Table saving of character-level: $38.4 M - 0.768 M = 37.63 M$ params, i.e. $\frac{0.768}{38.4} = 0.02$ , so the character table is only $2%$ as big. Why this step? The fixed saving from the tiny table is $\approx 37.6 M$ params — a one-time win to weigh against the recurring attention cost.
Weigh them: character-level saves a fixed $37.6 M$ params but pays $\approx 8 \times$ attention every forward pass. Why this step? Fixed saving vs. recurring $8 \times$ cost — for any real workload with many passes, the recurring side dominates.

Verify: $2.85 7^{2} = 8.16$ ✓ and $\frac{1000}{50000} = 0.02$ ✓, and $38.4 M - 0.768 M = 37.632 M$ ✓. Decision: subword wins for Chinese — matching the parent note's conclusion — because Chinese characters are dense and merging them cuts $n$ hard. See 6.2.01-Multilingual-models and 4.2.02-Byte-pair-encoding for how those merges are learned.

The figure below draws this exact tug-of-war: the magenta bars (left axis) show the attention-score cells — tall for character-level, short for subword — while the violet bars (right axis) show the embedding-table size, which is the opposite way round. Read it as "each tokenizer is short on one axis and tall on the other."

Worked example Example 8 — The seesaw, drawn (synthesis of Cells A + B)

[Cell A ∩ B, geometric] Plot how total cost behaves as $V$ climbs: embedding cost rises linearly, attention cost falls (because $n$ shrinks as tokens merge). Find, qualitatively, the sweet spot. This example is the geometric synthesis of the table cost (Cell A) and the attention cost (Cell B) — showing how the two prices fight as $V$ moves.

Forecast: guess the shape of total cost vs $V$ — monotone down? monotone up? or a valley?

Left arm — embedding cost $= V \times d$ : a straight rising line in $V$ . Why this step? Established in Example 1; nothing about the text changes it, so as $V$ grows this term climbs without bound.
Right arm — attention cost $\propto n^{2}$ , and $n$ drops as $V$ grows (bigger vocab → longer average token $\overset{ˉ}{ℓ}$ → fewer tokens $n = C / \overset{ˉ}{ℓ}$ ). So this curve falls and then flattens. Why this step? From the definition block, $\overset{ˉ}{ℓ}$ rises sublinearly with $V$ , so $n$ shrinks with diminishing returns — the fall steepens at small $V$ and levels off at large $V$ .
Sum them: rising line + falling-and-flattening curve $=$ a U-shaped valley. The bottom of the valley is the goldilocks vocab size $V^{⋆}$ . Why this step? A function that is large at both ends (attention-dominated on the left, table-dominated on the right) and lower in between must have an interior minimum. That minimum is exactly where "too small wastes compute, too large wastes memory" balances out — the parent note's central tension, made a single point on a curve.
Read the takeaway: to the left of $V^{⋆}$ you should raise $V$ (attention is the bottleneck); to the right you should lower $V$ (the table is the bottleneck). Why this step? The slope of the total curve tells you which way to move: downhill toward the valley floor. This turns a vague "goldilocks" slogan into an actionable direction.

Verify: At the low- $V$ end attention dominates (Example 4's $16 \times$ blow-up); at the high- $V$ end embedding dominates (Example 5's $3 B$ table). A quantity high on both ends and lower in the middle must dip to an interior minimum — a valley. ✓ The conclusion is complete: total cost is U-shaped, and the optimum $V^{⋆}$ sits at the bottom where the rising table cost and falling attention cost cross in influence.

The figure below plots all three curves against $V$ : the violet rising line is embedding cost, the orange falling curve is attention cost, and the magenta curve on top is their sum. The navy dot marks the valley floor — the goldilocks $V^{⋆}$ where total cost is least.

Worked example Example 9 — Exam twist: the hidden false assumption

[Cell H] "Claim: switching from $V = 32000$ to $V = 64000$ halves the sequence length, so attention cost quarters. True or false?"

Forecast: guess before reading — does doubling $V$ double $\overset{ˉ}{ℓ}$ ?

Spot the buried assumption: "doubling $V$ halves $n$ " secretly assumes $\overset{ˉ}{ℓ}$ doubles (since $n = C / \overset{ˉ}{ℓ}$ , halving $n$ requires doubling $\overset{ˉ}{ℓ}$ ). Why this step? Exam twists hide a false proportionality. Naming it explicitly is what lets you test it.
Check against reality: the parent note states $\overset{ˉ}{ℓ}$ grows sublinearly — doubling $V$ raises $\overset{ˉ}{ℓ}$ by only $\approx 20 - 30%$ , not $100%$ . Why this step? Empirically $\overset{ˉ}{ℓ} \approx 4$ at $32 K$ and $\approx 5$ at $100 K$ ; that is a $\approx 25%$ rise over a $\approx 3 \times$ vocab jump — nowhere near doubling.
Correct estimate: if $\overset{ˉ}{ℓ}$ rises $\approx 20%$ (from 4.0 to 4.8), then $n$ falls to $\frac{4.0}{4.8} = 0.833$ of before, and attention cost to $0.83 3^{2} \approx 0.694$ — a $\approx 31%$ cut, not $75%$ . Why this step? Apply the $n^{2}$ law to the true length change, not the assumed one.

Verify: $0.833 3^{2} = 0.694$ , i.e. attention drops to $\approx 69%$ (a 31% saving), decisively refuting the "quarters" claim. False. ✓ The trap was linear thinking about a sublinear relationship — see 4.2.03-WordPiece-and-SentencePiece for why merges saturate.

Recall Self-test

Attention score memory carries the $d$ dimension. True or false? ::: False — scores are $n \times n$ scalars; only $Q, K, V$ projections carry $d$ . Doubling sequence length changes attention cost by what factor? ::: $\times 4$ (it scales as $n^{2}$ ). Embedding table size depends on the text being processed. ::: No — it is $V \times d$ , fixed by vocab and width, independent of any input. Why does a bigger vocab not proportionally shorten sequences? ::: Average token length $\overset{ˉ}{ℓ}$ grows only sublinearly with $V$ . In the U-shaped total-cost curve, what lives on the two arms? ::: Rising arm = embedding cost ( $V d$ ); falling arm = attention cost ( $\propto n^{2}$ ). When you compare a fixed param cost to a per-batch activation cost, what must you do first? ::: Convert both to the same unit (bytes), since params and activation scalars are only comparable once counted in bytes.

See also: Vocabulary size tradeoffs · 4.3.01-Transformer-attention-complexity · 4.5.02-Embedding-layer-design · 5.1.03-Memory-optimization-techniques · 4.2.04 Vocabulary size tradeoffs (Hinglish)