Question bank — Mixture-of-Experts (MoE) architecture

False — per-token compute scales with the active count $k$, not the total $N$. Adding experts grows storable knowledge (memory/capacity) while leaving the $k$ evaluated experts unchanged.

False — it is roughly as fast as a dense model with only the active parameters (a few experts + shared layers), which can be an order of magnitude smaller than 448M.

True after re-normalization — with a single selected expert, $\mathcal{T}$ (the set of chosen experts) has one element so $g_i=p_i/p_i=1$. But that "1" still carries the router's gradient, so the router keeps learning even at $k=1$.

False — because the gate $g_i$ multiplies the expert output $y=\sum g_i{E}_i(x)$, the main task loss flows through $g_i$ into the router matrix $W_g$. The auxiliary loss only adds a balancing pressure on top.

False — routing is per token (even per position). "Bank" in one sentence and "bank" in another can land on completely different experts; that per-token freedom is where specialization comes from.

False — a dropped token skips the expert computation but still passes through via the residual connection, so it carries its input forward unchanged rather than vanishing.

False — it multiplies $f_i$ (fraction of tokens dispatched to expert $i$) by the soft average probability $P_i$ (mean router probability for expert $i$): ${\mathcal{L}}_{\text{aux}}=\alpha N{\sum }_i{f}_i{P}_i$. The $P_i$ factor is differentiable, so gradients can actually reshape the routing; $f_i$ from an argmax alone is non-differentiable.

False — it only reduces the chance of overflow by raising each expert's cap $C$. A pathologically skewed router could still exceed even a generous $C$; capacity factor trades memory/compute against drop risk, not a hard guarantee.

Error: for $k\ge 2$ you must blend outputs, so you need weights summing to 1; and softmax's smooth gradient is what lets the router learn — a bare argmax is non-differentiable.

Error: the whole point is to never evaluate the unselected experts. $g_i=0$ conceptually zeroes them, but in practice only the $k$ chosen experts are ever run — that omission is the FLOP saving.

Error: an expert trains whenever it is selected for some token in the batch. Over many tokens most experts get selected sometimes, so all of them accumulate gradient — just not from every token.

Error: the $N$ is deliberate. At the uniform optimum $f_i=P_i=1/N$, the sum is $N\cdot (1/N{)}^2=1/N$, and multiplying by $N$ makes the loss value $\approx 1$ independent of expert count, so $\alpha$ has a consistent meaning across model sizes.

Error: the raw $p_i$ over $\mathcal{T}$ (the selected-expert set) generally sum to less than 1, so the output would be systematically scaled down. Re-normalizing keeps the gate weights a proper convex blend summing to 1.

Error: each expert is an independent FFN. MoE replaces one dense FFN with many FFN experts plus a router — it multiplies FFNs, it doesn't delete them.

Here $W_g$ is the router's weight matrix and $h_i$ the pre-score for expert $i$. A dot product measures alignment between the token $x$ and each expert's learned preference row of $W_g$; it is linear, cheap, and pairs naturally with softmax to turn "who aligns best" into normalized probabilities $p_i$.

Because a "rich-get-richer" feedback loop makes the router collapse onto a few experts, leaving most parameters dead. The main loss doesn't penalize this, so an explicit auxiliary loss is required to keep experts busy — without it the entire capacity advantage is lost.

Model quality tracks parameters known, while cost tracks FLOPs per token. Dense models tie these together; MoE lets you climb the parameter axis of the scaling curve while holding the compute axis roughly fixed.

The balancing loss is a gentle nudge, not the objective. Too large an $\alpha$ would force uniform routing that ignores what each token actually needs, hurting quality; too small and collapse creeps back.

The hard selection determines which experts run, but the softmax-derived gate weight $g_i$ gives a differentiable multiplier, letting gradients teach the router which expert should have been chosen and by how much.

The layer degenerates into a dense mixture: every expert runs, the selected set $\mathcal{T}$ is all $N$ so re-normalization does nothing new, and you lose all the FLOP savings — you've paid for capacity without conditional computation.

The framework applies a deterministic tie-break (e.g. lower index wins). Only $k$ can enter $\mathcal{T}$, so one of the tied experts is dropped for that token — a reminder that Top-$k$ is a hard, discrete cut.

Top-$k$ still picks $k$ experts (by tie-break), each getting gate $g_i=1/k$ after re-normalization. The token gets a uniform blend of $k$ experts — no specialization signal, but a valid output.

Routing is near-random, so tokens scatter roughly uniformly — which is actually healthy. The danger is later, once a small skill gap triggers the rich-get-richer collapse; that's precisely when the aux loss earns its keep.

Then that token effectively loses this layer's transformation and passes through near-unchanged, degrading quality. This is why drop rate is monitored and capacity factor tuned — persistent drops silently hollow out the model.

$C$ falls below the average tokens-per-expert, so even perfectly balanced routing overflows and drops tokens every batch. Capacity factor $\ge 1$ is the floor for lossless routing under balance; $<1$ deliberately trades quality for compute/memory savings.