Mixture of Experts Architecture
HardConceptualMoEarchitectureefficiency
Related module: Training and scaling language models
Problem statement
Mixture of Experts (MoE) models activate only a subset of parameters for each input.
- How does the MoE architecture work?
- What is the routing mechanism and its challenges?
- Compare MoE vs dense models in terms of compute and quality.