Sparse routing and gating networks: Question bank

This is a self-test bank for Sparse routing and gating networks. Each line is a Question ::: Answer reveal. Cover the answer, commit to a response out loud with justification, then reveal. If your reason differs from the given one — even when your yes/no matches — treat it as a miss and revisit the parent note.

True or false — justify

Adding more experts

N

(with

k

fixed) increases the compute cost per token.

False. Active compute is

O (k \cdot d \cdot d_{ffn})

— it depends on

k

, not

N

. Growing

N

adds parameters (and a tiny

O (N \cdot d)

gating cost) but each token still runs through only

k

experts.

A Mixture-of-Experts layer has more total parameters than a dense layer but similar active compute.

True. That is the entire selling point: massive parameter store, constant per-token FLOPs. This is why MoE is a form of Conditional Computation.

The softmax in the gating network is there mainly to make routing sparse.

False. Softmax makes scores into a differentiable probability distribution so gradients can flow; sparsity comes afterwards from the separate top-

k

selection step.

If the gating network sends every token to the same expert, training will still converge to a good balanced model on its own.

False. This is expert collapse — the favoured expert improves, gets chosen more, and the others starve. Without a load-balancing pressure the imbalance is self-reinforcing.

The coefficient of variation

CV = σ / μ

of the load is what the Switch Transformer actually minimizes during backprop.

False. CV is a monitoring metric only — it is computed through the discrete top-

k

and is not differentiable. The optimized surrogate is

L_{aux} = α N \sum_{i} f_{i} P_{i}

In the Switch auxiliary loss

α N \sum_{i} f_{i} P_{i}

, gradients flow through both

f_{i}

and

P_{i}

False. Only

P_{i}

(mean gate probability) is differentiable;

f_{i}

(fraction of tokens dispatched) comes from the hard top-

k

and is treated as a detached constant.

Expert Choice routing still needs an auxiliary load-balancing loss.

False. In Expert Choice each expert grabs exactly its top-

C

tokens, so balance is structural — guaranteed by construction, not learned.

Soft MoE removes the discontinuity of top-

k

by having experts process weighted mixtures of all tokens.

True. There is no arg-max or top-

k

anywhere, so the whole layer is differentiable and balanced by construction, at the cost of computing

M = N \cdot S

slot mixtures.

Switch Transformer's

k = 1

means each token gets less per-token capacity than a

k = 2

router.

True. One expert per token is maximally sparse — cheaper and easier to balance, but less capacity blended per token. Switch trades this off by scaling

N

to 1000+.

The gating network sees gradient signal about experts it did not select.

False (for vanilla top-

k

). Non-selected experts contribute nothing to the output, so they and their gate scores get zero gradient — the source of the "broken gradients" trap in the parent note.

Spot the error

"With

N = 128

and

k = 2

, we store 2 experts and simulate the other 126."

Backwards. We store all 128 experts (full parameter capacity) and only compute 2 of them per token. Sparsity is in the compute, not the storage.

"The auxiliary loss is minimized when one expert takes all the tokens, since that's the clearest routing."

Wrong.

\sum_{i} f_{i} P_{i}

is minimized (value

= 1

) exactly at uniform routing,

f_{i} = P_{i} = 1/ N

. Concentration on one expert makes the product large.

"Expert Choice guarantees every token gets processed by exactly

k

experts."

Not guaranteed. It guarantees every expert processes exactly

C

tokens. A low-affinity token may be picked by fewer than

k

experts (or dropped), because experts choose tokens, not the reverse.

"Soft MoE's dispatch matrix

Φ

has shape

B \times S

because there are

S

slots per expert."

Wrong shape. Each expert owns its own block of

S

slots, so total slots are

M = N \cdot S

and

Φ

B \times (N \cdot S)

. Every expert needs its own dedicated slot columns.

"Renormalizing the top-

k

gates (dividing by their sum) changes which experts are selected."

No. Renormalization happens after TopK on the already-chosen set — it rescales weights so they sum to 1 but never alters membership of the selected set.

"Because gating cost is

O (N \cdot d)

, doubling

N

roughly doubles the layer's total FLOPs."

Wrong. Gating cost is tiny compared to expert FFNs

O (k \cdot d \cdot d_{ffn})

; doubling

N

barely moves total active compute, which is dominated by the

k

experts.

"Switch's single

f_{i} P_{i}

loss and Shazeer's two squared-CV terms optimize opposite objectives."

Wrong. Both push toward uniform expert usage; Switch's product form is just a simpler, later refinement of the same balancing idea.

Why questions

Why do we apply softmax before top-

k

rather than just picking the largest raw logits?

The softmax gives a normalized, differentiable weight to each selected expert so the combination

\sum G (x)_{i} E_{i} (x)

is a proper convex mix and gradients to the gate are well-scaled; raw logits have no bounded scale.

Why is CV a good diagnostic even though it's a bad loss?

It is scale-invariant and hits exactly 0 when all loads are equal, making it a clean human-readable imbalance gauge — but it is computed through the non-differentiable top-

k

, so you cannot backprop through it.

Why does the Switch loss carry the factor

N

It keeps

L_{aux} = O (1)

regardless of how many experts you have. Without it the sum of

N

tiny

\frac{1}{N} \cdot \frac{1}{N}

terms would shrink as you add experts, weakening the balancing pressure.

Why does MoE pair naturally with Model Parallelism?

Different experts are independent sub-networks that can live on different devices; a token only activates

k

of them, so you shard experts across hardware and route tokens to the device holding their chosen expert.

Why can a token be dropped in token-choice routing but (ideally) not in Expert Choice?

Token-choice experts have a fixed capacity

C

; if too many tokens pick the same expert, the overflow is dropped. Expert Choice fills each expert to exactly

C

by construction, so capacity is never overrun.

Why does the parent note call the hospital receptionist a metaphor for the gating network and not the experts?

The receptionist only routes — decides which specialist to send you to — mirroring

G (x)

producing routing scores. The specialists (experts

E_{i}

) do the actual work.

Why is sparse routing relevant to modelling a Long-tail Distribution of inputs?

Rare input types can get their own specialist expert that only fires for them, giving the tail dedicated capacity without inflating the compute paid on common inputs.

Edge cases

N = 1

(single expert): what does the MoE layer reduce to?

A plain dense FFN. With one expert,

G (x) = 1

always, top-

k

is trivial, and

y = E_{1} (x)

— there is nothing to route.

k = N

(select all experts): is this still "sparse" routing?

No. Every expert fires for every token, so active compute is

O (N \cdot d \cdot d_{ffn})

— a dense weighted ensemble, the opposite of the sparsity we wanted.

Two experts tie for the top score in Switch (

k = 1

): what happens?

The arg-max must break the tie by a fixed rule (e.g. lowest index). It is a measure-zero event in continuous logits but must be handled deterministically for reproducible routing.

A batch where all tokens legitimately want the same expert: does the balancing loss hurt accuracy?

Potentially yes — the auxiliary loss will pressure the gate away from the "correct" concentration. That is why

α

is kept small: balance is a soft regularizer, not a hard constraint.

Expert Choice with capacity

C = B k / N

not an integer: what must you do?

Round

C

(typically down or via a capacity factor) — a fractional slot count is meaningless, so implementations floor/ceil

C

and accept slight over/under-provisioning.

Empty batch or

B < N

tokens in Expert Choice: can every expert fill

C

slots?

No. If there are fewer tokens than total slots, some experts sit idle or reuse tokens;

C = B k / N

can be below 1, so the "exactly

C

tokens each" guarantee degrades at tiny batch sizes.

Gate probabilities nearly uniform (untrained network at step 0): why is this actually the desired balanced starting point but still a problem?

Uniform gates mean maximal balance (good) but zero specialization — experts are interchangeable, so early training must break symmetry before any expert can become a useful specialist.

Recall One-line summary to lock in

Sparse routing = full parameter store, $k$ -expert compute; softmax for gradients, top- $k$ for sparsity, and a detached- $f_{i}$ , differentiable- $P_{i}$ auxiliary loss (or structural Expert-Choice / differentiable Soft-MoE) to stop expert collapse.