Mixture of experts
A mixture-of-experts (MoE) layer replaces one big MLP with many expert MLPs, plus a router that sends each token to only a few of them. The model has many parameters, but uses only a fraction of them for each token.
Why it matters
Bigger models know more, but every parameter normally runs for every token, so cost grows with size. MoE breaks that link: total knowledge can grow while compute per token stays modest. Many frontier models use it, and it changes how you think about model size, memory and serving.
The idea
A mixture-of-experts (MoE) layer replaces one big MLP with many smaller 'expert' MLPs and a router. For each token, the router picks only a few experts, often 2, to run. So the model can have a huge number of parameters in total while each token uses only a small part of them: more knowledge for about the same compute per token.
A hospital with many specialists. Each patient sees only the one or two doctors they need, not every doctor in the building.
A hospital can spread patients fairly. In an MoE, if the router sends too many tokens to the same few experts, they are overloaded and the others learn nothing, so training adds a load-balancing loss. And every expert must still sit in memory, even the unused ones.
How it works
- Inside each block, replace the single MLP with E experts (for example 8), each an MLP of its own.
- A small router (a linear layer) scores every expert for each token.
- Keep the top k experts (often 2), turn their scores into weights with softmax, and run only those experts.
- The token's output is the weighted sum of its chosen experts' outputs.
- Different tokens choose different experts, so experts can specialise.
- A load-balancing loss pushes the router to spread tokens evenly; without it a few experts get everything and the rest never learn.
- All experts must be in memory even though each token uses only some of them, so memory scales with the total size while compute scales with the active size.
Key points
- Total parameters ≠ active parameters. Mixtral 8x7B: about 47B total, about 13B active per token.
- Top-k routing: each token is sent only to its k best-scoring experts, usually 1 or 2.
- Load balancing is essential: without it the router sends almost every token to a few experts, and the rest never learn.
- A mixture of experts saves compute, not memory: every expert must still be loaded, although each token uses only a few.
- Experts across GPUs need fast communication (expert parallelism).
Common mistakes
- Quoting the total parameter count as if it were the compute cost per token.
- Training without a balancing loss and getting "dead" experts.
- Forgetting that serving still needs memory for every expert.
Not true: A mixture-of-experts model needs memory only for the experts that a token actually uses.
In fact: Every expert must be loaded, because the next token may be sent to a different one. The saving is in compute per token, not in memory.
Not true: Routing: each token passes through the full set of experts.
In fact: A router sends each token to only its k best-scoring experts, usually 1 or 2. That is why the active parameters are far fewer than the total.
Part of Unit 4: Transformers and LLMs in the free ML Foundations course.