Question bank — Load balancing in MoE

False. Under only ${\sum }_i{f}_i=K$ and ${\sum }_i{P}_i=1$, the raw dot product is smallest at a corner — dump all $f$-mass on the smallest-$P$ expert (see the intuition callout above). Uniformity is a training fixed point, not the analytic minimizer of the bare product.

False. Too large an $\alpha$ over-weights balance versus the task loss ${\mathcal{L}}_{\text{task}}$, forcing near-uniform routing that ignores which expert is actually best for a token, hurting quality. It's a trade-off, hence typical $\alpha \approx 0.01$.

False. $f_i$ is a hard indicator count and is treated as constant; $\partial {\mathcal{L}}_{\text{aux}}/\partial P_i=\alpha N{f}_i$. The correction happens entirely through the differentiable $P_i$, scaled by the measured load $f_i$.

False. It settles to a small positive constant $\alpha K$: with $f_i=K/N$ and $P_i=1/N$, each of the $N$ terms is $(K/N)(1/N)$, so $\alpha N{\sum }_i(K/N)(1/N)=\alpha N\cdot N\cdot \frac{K}{N^2}=\alpha K$. Balance minimizes it, it does not zero it.

False. The $\alpha$ is already inside ${\mathcal{L}}_{\text{aux}}=\alpha N{\sum }_i{f}_i{P}_i$. Writing it again applies $\alpha$ twice — the correct form is simply ${\mathcal{L}}_{\text{task}}+{\mathcal{L}}_{\text{aux}}$.

False. $C$ sets each expert's buffer size ($C\cdot B/N$); a larger $C$ drops fewer tokens but does nothing to make the router choose experts evenly. It hides imbalance's symptom, not its cause.

False. It's a natural feedback loop: a slightly-better expert attracts more tokens, gets more gradient, improves faster, attracts even more. The softmax is behaving correctly — the dynamics need a counter-force (the aux loss).

True. Softmax outputs are a probability distribution summing to 1, so mass removed from an overloaded expert must reappear across the others. How it redistributes depends on the pre-softmax logits — softmax spreads the freed mass proportionally to the other experts' exponentiated scores, not equally, so idle-but-slightly-favoured experts gain the most.

False. $f_i$ is a non-differentiable hard count, so ${\sum }_i{f}_i^2$ has zero gradient through the router — it can't teach ${\mathbf{W}}_g$ anything. You need the differentiable $P_i$ in the product.

Wrong framing. Here anti-correlation would mean the loss symmetrically rewards making $f$ and $P$ disagree (big $f$ paired with small $P$ and vice-versa) on both variables. But $f_i$ is frozen as a scaling constant and only $P_i$ moves: the gradient $\alpha N{f}_i$ pushes $P_i$ down in proportion to current load — a one-sided feedback controller, not a symmetric anti-correlation objective.

Error: confusing tokens with the constraint. ${\sum }_i{f}_i=K$ because $f_i$ is a fraction (divided by $B$); each token contributes $K$ picks spread over the fractions, summing to $K=1$, not $8$.

Error: wrong reason. Dropping is a deliberate capacity constraint creating back-pressure — over-routed experts overflow, those tokens skip the expert (pass as residual), and the resulting worse performance discourages over-routing. It's a training signal, not an OOM handler.

Error: it uses the renormalized ${\tilde{g}}_i$. After picking Top-K, weights are re-normalized so the chosen experts' gating scores sum to 1; the raw $g_i$ over all $N$ experts do not sum to 1 within the selected set.

Error. It's an auxiliary term added to ${\mathcal{L}}_{\text{task}}$. The router optimizes both simultaneously — good task routing plus balanced load — never one instead of the other.

${\sum }_i{P}_i^2$ regularizes only the router's confidence, ignoring the actual dispatched load — which is what causes hardware and compute imbalance. The cross term lets the measured imbalance $f_i$ scale the corrective push on the differentiable $P_i$.

Because $\partial {\mathcal{L}}_{\text{aux}}/\partial P_i=\alpha N{f}_i$: the more overloaded an expert (bigger $f_i$), the harder the loss pushes its probability down. The correction is exactly proportional to how badly balanced that expert is.

Each token picks $K$ experts, so total dispatch fraction is $K$, spread evenly gives $K/N$ per expert. For Top-2 of 8 experts that's $2/8=25\%$, not $1/8$.

A low $C$ shrinks each expert's buffer, so more tokens overflow and get dropped; dropped tokens skip their best expert and get a worse representation, degrading accuracy. Typical $C\in [1.0,1.5]$ balances balance versus dropping.

Starved experts receive almost no gradient signal, so they never improve, so the router never finds a reason to send them tokens — a self-reinforcing dead zone the aux loss must break.

Exactly $K=1$, since each token is a single pick and $f_i$ are fractions of the batch. This is independent of batch size.

With $K=0$ every token skips all experts, so every $f_i=0$ and the loss is $\alpha N{\sum }_{i}0\cdot P_i=0$. There's nothing to balance because nothing is dispatched — a vacuous, non-functional MoE layer, which is why real routing always uses $K\ge 1$.

Much larger: $f_1=1$ concentrates the whole product on the busiest expert's $P_1$, and the gradient $\alpha N\cdot 1$ slams $P_1$ down hard — the loss is maximally above the balanced floor of $\alpha K$.

It settles to $\alpha K=2\alpha$ (e.g. $0.02$ for $\alpha =0.01$) — a small nonzero constant, since balance minimizes but never zeros the term.

The auxiliary term vanishes entirely, the router optimizes only ${\mathcal{L}}_{\text{task}}$, and the collapse feedback loop is left unchecked — a few experts typically dominate.

It overflows and is dropped for that expert — its representation passes through as a residual, bypassing the expert computation entirely for that token.

Not instantly and not guaranteed — collapsed experts are hard to revive because they carry no useful gradient; balancing works best as a preventive force applied from the start, alongside capacity-based back-pressure.