Deep Dive into Mixture of Experts: From 1991 to DeepSeek-V3
Every major LLM lab is in a conundrum today, deliberating between scale vs cost. Making a dense model bigger makes it smarter yes, but also makes every token more expensive to generate. In a dense model, every parameter illuminates on every token, and the compute cost of a forward pass scales ~linearly with parameter count. Mixture of Experts (MoE) challenges the status quo with its architecture.…
In 1991, a paper titled "Adaptive Mixtures of Local Experts" by Jacobs, Jordan, Nowlan, and Hinton introduced the concept of mixture of experts (MoE). The idea was to train multiple smaller expert networks alongside a gating network that learned how to weight and combine their outputs. While this approach did not reduce compute costs, as all experts ran on every input, the use of a learned, differentiable routing function was a conceptual ancestor to modern LLM MoE.
In 2017, Google Brain's Shazeer et al. paper introduced sparse gating, a key innovation that significantly improved efficiency. Instead of every expert computing on every input, the gate selected only the top-k highest-scoring experts (4/4096 in this case), activating only those experts for a given input. This sparse gating mechanism saved compute costs and became the foundation for modern LLMs with MoE architecture.
Subsequent developments in 2020-2021 included Google's GShard (2020) and Switch Transformer (2021), which integrated sparse MoE into Transformer architectures. GShard used top-2 gating for distributed sparse training at scale, while Switch Transformer simplified routing to top-1 gating, making the model cheaper and more stable to train.
These advancements paved the way for the development of large-scale, efficient LLMs like DeepSeek-V3, which utilizes MoE to achieve remarkable performance while managing computational resources effectively.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
