Explaining Mixture of Experts (MoE) Architecture in Plain English

A lot of people have heard that GPT-4 and Mixtral use Mixture of Experts architecture, but explanations online tend to be very technical. Let me try to explain it simply.

Imagine you have a company with 8 specialized consultants: one knows finance, one knows marketing, one knows engineering, etc. When a question comes in, a “router” reads the question and decides which 2 consultants should answer it. Only those 2 do the work – the other 6 sit idle.

That’s basically MoE. The model has multiple “expert” neural networks, and a gating mechanism selects which experts to activate for each piece of input. This means:

– The total model can be huge (trillions of parameters)
– But inference only uses a fraction of those parameters per query
– So it’s both more capable AND more efficient than a dense model of the same compute budget

Why it matters: MoE is how companies make increasingly powerful models without proportionally increasing compute costs. It’s a major reason why we can get GPT-4 quality responses at reasonable speeds.

The tradeoff: MoE models need more memory (all experts must be loaded, even if only 2 are active) and can be harder to train stably.

Anyone have corrections or additional details?