What is a mixture of experts (MoE) model?
1 Answer
A mixture of experts, or MoE, is an AI model design where the model contains many specialised sub-networks, called experts, but uses only a few of them for each piece of input. This lets a model have a very large total size, holding lots of knowledge, while staying faster and cheaper to run than a similarly sized traditional model.
How does it work? In a traditional dense model, every parameter is used for every token processed. In an MoE model, certain layers contain multiple expert networks plus a component called a router or gating network. For each token, the router decides which experts are most relevant and sends the token to only a few of them, typically two or a small number out of many. The outputs of the selected experts are combined.
An analogy: think of a large hospital with many specialist doctors. A patient is not examined by every doctor; a reception desk directs them to the most relevant specialists. The hospital has broad expertise overall, but each patient's visit uses only a few doctors' time.
Key terms:
Total parameters: all parameters in the model, including every expert.
Active parameters: the parameters actually used for each token.
For example, a model might have hundreds of billions of total parameters but only tens of billions active per token. It gains knowledge capacity from the large total size while computing costs depend mainly on active parameters.
Advantages:
1. Efficiency: faster and cheaper inference than a dense model of the same total size.
2. Scalability: easier to grow model capacity without proportional increases in computing per token.
3. Strong performance: MoE models have achieved excellent results for their computing cost.
Disadvantages:
1. Memory: all experts must usually be stored in memory, even if only some are used, so MoE models need a lot of memory to run.
2. Training complexity: balancing work among experts and training the router properly is tricky.
3. Deployment complexity, especially across multiple GPUs.
Where is MoE used? Mistral's Mixtral was a well-known open MoE model. DeepSeek's models used MoE designs to achieve strong results efficiently, which was part of why they attracted attention. Many open models from Qwen and others use MoE, and several closed frontier models are widely believed to use similar approaches, though companies do not always disclose architecture details.
For people running models locally, remember that an MoE model's memory requirement depends on total parameters, while its speed depends more on active parameters.