Mixture of Experts (MoE)
An architecture that routes each token to a small subset of specialized expert networks.
MoE lets models have huge parameter counts while only activating a fraction per token, drastically cutting inference cost. Mixtral, DeepSeek and GPT-4 all use MoE.