Foundations — Mixture-of-Experts (MoE) architecture

This page assumes nothing. Every letter, arrow, and word in the parent note is unpacked below, in an order where each idea only leans on the ones before it. If a symbol confused you upstairs, its definition lives here.

0. The very first picture: a token and its vector

Before any experts exist, we need to know what a network sees.

A network cannot do arithmetic on the letters "c-a-t". So each token is turned into a list of numbers.

Look at the figure: the word "bank" becomes an arrow in space. Its direction encodes meaning. Two tokens that mean similar things point in similar directions. This "direction = meaning" idea is the seed of everything — the router will judge experts by how well their direction matches the token's direction.

1. The dot product — "how aligned are two arrows?"

The router's whole job is to ask: does this token point the same way as expert $i$'s preferences? The tool that answers "how aligned?" is the dot product.

Why this tool and not another? We need to squash two arrows into one "match score". The dot product is exactly that: it is large-and-positive when the arrows point the same way, near zero when they are perpendicular (unrelated), and negative when they point opposite ways.

Look at the three panels: same arrows (big score, green), right-angle arrows (zero, yellow), opposing arrows (negative, red). The router uses this to say "expert 3's preference vector lines up with this token — send it there."

2. The router weight matrix $W_g$ — one preference vector per expert

Each expert wants its own "what tokens do I like?" arrow. Stacking $N$ such arrows gives a matrix.

Why a matrix? Because doing $N$ separate dot products is tedious; the matrix packs "score all experts at once" into one clean operation. That is all $W_{g}x$ is — $N$ alignment scores in a stack.

3. $e$ and the exponential — turning any number into a positive one

The scores $h_i$ can be negative. To treat them as weights we first make them all positive, keeping their order. The tool is the exponential function.

Look at the curve: feed in $h=-2$ and you still get a small positive number ($\approx 0.14$); feed in $h=3$ and you get a big one ($\approx 20$). Ordering is preserved: the biggest logit stays the biggest.

Why exponential and not, say, "add 100 to everything"? Because $e^h$ (a) guarantees positivity for all inputs including very negative ones, and (b) is smooth — you can take its slope everywhere, which the learning process (gradients) needs.

4. Softmax — from scores to a probability blend

Now we have positive numbers. To compare them fairly and later use them as weights, we need them to sum to 1 (a proper blend, no leftover). That normalization is softmax.

Why do we need "sum to 1"? Because later (in §5–§6) these numbers become the weights that blend the chosen experts' outputs. Weights that sum to 1 give a safe weighted average — a convex combination — instead of a total that could blow up or shrink arbitrarily. We meet those weights and the blend itself next.

Softmax and this exact reasoning are explored further in Softmax and Gating Functions.

5. Top-$k$ and the index set $\mathcal{T}$ — keep the few, zero the rest

Softmax still gives every expert a nonzero slice. Running all $N$ experts is exactly the cost we are trying to avoid. So we keep only the $k$ biggest.

Now the weights themselves.

6. The experts $E_i$ and the output $y$

7. The counting symbols: $N$, $k$, $d$, and FLOPs

8. The residual connection — the "skip road" around the layer

Before we can talk about dropping tokens, we need the escape route they take.

Why does MoE need it? Because with limited expert capacity (next section) some tokens get no expert. The residual guarantees those tokens still have a value to pass forward — they are skipped, not deleted.

9. Balancing symbols: $f_i$, $P_i$, $\alpha$, and capacity $C$

These appear in the load-balancing helper loss; here is what each means in plain words.

Why two different "how busy is expert $i$" numbers ($f_i$ and $P_i$)? $f_i$ comes from a hard pick and cannot be nudged by gradients; $P_i$ is smooth and can be. The helper loss multiplies them so gradients can flow — the full story is in Load Balancing and Auxiliary Losses. Spreading experts across machines is Model Parallelism & Expert Parallelism.

Prerequisite map

Every arrow says "you must understand the left box to understand the right box." The topic MoE sits at box K, fed by all foundations.

Equipment checklist

Test yourself — cover the right side, answer, then reveal.